[GRADLE-2034] Existence of pom file requires that declared artifacts can be found in the same repository Created: 06/Jan/12  Updated: 15/Sep/13  Resolved: 09/Sep/13

Status: Resolved
Project: Gradle
Affects Version/s: None
Fix Version/s: 1.9-rc-1

Type: Bug
Reporter: Luke Daley Assignee: Daz DeBoer
Resolution: Fixed Votes: 20

Issue Links:
Duplicate
Duplicates GRADLE-2709 Dependency in Maven local with POM an... Resolved
Duplicated by GRADLE-2369 Dependency resolution fails for maven... Resolved

 Description   

This is easy to reproduce with m7.

Create a java project that pulls from mavenLocal(). Make sure hamcrest-core-1.1 is in your mavenLocal, rhen remove .m2/repository/org/hamcrest/hamcrest-core/1.1/hamcrest-core-1.1.jar.

If you run the testClasses task you'll get:


12:07:12.343 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.IvyResolverBackedArtifactToFileResolver] Attempting to download org.hamcrest#hamcrest-core;1.1!hamcrest-core.jar using resolvers [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.DependencyResolverAdapter@1586b489]
12:07:12.343 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.IvyLoggingAdaper] 		tried /Users/ld/.m2/repository/org/hamcrest/hamcrest-core/1.1/hamcrest-core-1.1.jar
12:07:12.344 [DEBUG] [org.gradle.api.internal.artifacts.repositories.ResourceCollectionResolver] Loading /Users/ld/.m2/repository/org/hamcrest/hamcrest-core/1.1/hamcrest-core-1.1.jar
12:07:12.344 [DEBUG] [org.gradle.api.internal.artifacts.repositories.ResourceCollectionResolver] Resource not reachable for org/hamcrest#hamcrest-core;1.1: res=/Users/ld/.m2/repository/org/hamcrest/hamcrest-core/1.1/hamcrest-core-1.1.jar

If you remove ~/.m2/repository/org/hamcrest/hamcrest-core/1.1/ then everything will work (if you are also using mavenCentral()).

Wild guess: when seeing if mavenLocal has the dependency, we are testing for /Users/ld/.m2/repository/org/hamcrest/hamcrest-core/1.1/ instead of the actual jar file.



 Comments   
Comment by Daz DeBoer [ 07/Jan/12 ]

This is the expected behaviour: we treat repositories as self-contained, and only search for artifact files in the same repository that provided the metadata. I guess the issue is that we could make a better effort to try other repositories when one repository is found to be incomplete.

Comment by Luke Daley [ 09/Jan/12 ]

Hmm, I see. Tricky situation.

It kinda seems like we should, but perhaps we should treat this as low priority. When I first encountered it I thought it might be a serious regression, but I guess this scenario is kind of edge.

Comment by Adam Murdoch [ 09/Jan/12 ]

It'll be hard to distinguish between a meta-data only module and a module that is missing the jar.

Perhaps this should be part of our 'best' strategy, where a module with some artifacts is 'better' than one with the same version and no artifacts. With perhaps a warning that we're not using the first one we found.

Comment by Mauro Molinari [ 10/Jan/12 ]

This problem seems to be in some way related to what I've described here:
http://forums.gradle.org/gradle/topics/making_gradle_find_a_jar_in_a_flatdir_repo_when_it_is_missing_from_a_maven_repo
The problem is somewhat different, but Daz's comment confirmed my observations.

I think Gradle should look at other repositories when it can't find something in the repository that provided the metadata, as a fallback option. Otherwise I really don't know how I could solve the problem I outlined in the forum topic above.
Moreover, if this fallback search worked, I could also provide missing sources/javadoc in my flatDir repos.

Comment by Ryan Stewart [ 03/May/12 ]

This definitely isn't an edge case, and it's still there in rc-2. It's fairly normal for a local Maven repo to have directories like this in it with only a POM and no associated artifact. I don't yet know a definite series of steps you can take to reproduce it, but if Maven doesn't have to download a jar, it won't download it. That doesn't mean that it never downloaded the POM in order to discover the full dependency tree, though.

This issue is currently a problem for my team, where we have Maven and Gradle projects mixed together. At present, the only workaround we've found is to remove mavenLocal from our repository list so that it can always resolve artifacts from a remote repo, where the jar is never missing. More importantly, it means when we need to install a new SNAPSHOT locally, we have to either 1) locally modify the build script to add mavenLocal to the top of the list, or 2) deploy the new snapshot to the remote repo.

Bottom line: this is just normal behavior for Maven, and it makes mavenLocal useless.

Comment by Daz DeBoer [ 11/Jun/12 ]

Also http://forums.gradle.org/gradle/topics/1_0_rc_3_multiple_maven_repositories_could_not_resolve_all_dependencies

Comment by Ryan Stewart [ 13/Aug/12 ]

I commented over in the thread mentioned above. A workaround is suggested, but it doesn't work for local maven repos.

Is anybody paying attention to this issue, or is it effectively ignored? It's a really annoying problem and is one of the primary things holding back our adoption of gradle on a wider scale.

Comment by Daz DeBoer [ 13/Aug/12 ]

This issue is on the list to get fixed. It is not being ignored, but it is also not our highest priority.

Comment by Ryan Stewart [ 13/Aug/12 ]

Cool. Thanks for the reply.

Comment by Kurt Harriger [ 03/Nov/12 ]

Ran into this as well when working with mixed projects. I found that deleting .m2/repository seemed to resolve the issue but then it would return again later, I then discovered I could just comment out mavenLocal and it would build providing all the artifacts where uploaded to remote repo. Now that I know the issue and how to work around it not such a bit deal, but perhaps a short term improvement would be to add additional error messaging when using multiple repositories so that other users like me don't spend hours digging through dependency trees trying to isolate the problem only to have it start working then break again later next time you update a dependency.

Comment by Andrey Myatlyuk [ 04/Mar/13 ]

We have this issue and I still looking for the reason why artifacts are missing in .m2, do you build with Maven on the same node?

Comment by Eric Pantera [ 03/May/13 ]

Hi there. We were considering moving a 80 developers teams from Maven to Gradle, but this issue is a blocker point. It is just not acceptable to ask to everybody to remove artifacts from local repo or to do comment if necessary the mavenLocal().

Comment by Mauro Molinari [ 06/May/13 ]

I agree. I follow Gradle development with much interest, but this is one of the most important fixes that is still missing from it
In particular, my own use case (pom+binary jar in a remote repository + sources in a local flat dir repository, or jsut pom in a remote repository and jar in local flat dir for license issues) would solve a lot of headaches to us.

Comment by Fedor Belov [ 06/May/13 ]

It's so annoying to remove it from local maven repo again and again. Pls fix it!

ps
Does anybody know why local maven repo sometimes doesn't contain jars?

Comment by Ryan Stewart [ 06/May/13 ]

@Fedor: To my knowledge, it's because Maven might need to read a pom in order to discover correct transitive dependencies, but then that specific artifact doesn't end up in the final dependency graph, so it doesn't need to download it.

This still affects my team daily, and we've been dealing with it for a year now. It's even worse when you consider that Gradle still has no "local install" option besides borrowing the Maven plugin's capability, which forces you to use Maven local. <rant>How can this critical function (local artifact installation/resolution) still be missing from such an "evolved" build tool after more than a year?!?</rant>

Comment by Andrey Myatlyuk [ 06/May/13 ]

@Ryan
We use Gradle to publish to local Maven repository or remote Maven repository(Artifactory). Can you please explain your sentence "Gradle still has no "local install" option besides borrowing the Maven plugin's capability, which forces you to use Maven local". Did you have a chance to look at Maven Publish? http://www.gradle.org/docs/current/userguide/publishing_maven.html

@Fedor
It is only happening when you are using two build systems on your workstation and both are served from the same .m2 location. You can install Artifactory or Nexus, publish your artifacts there, or you can start using Gradle on all of your builds Also, you can resolve from one and publish to another local repository. There are many ways to go around it.

Comment by Fedor Belov [ 06/May/13 ]

@Andrey

I still think that maven is my default repository. I think there are a lot of people like me. You should fix it, because:
1. First time I met this problem I spent a lot of time to solve it
2. You can say that this is a feature but it looks like a bug - technically everything is ok, but it doesn't work. Don't think as developer. Think as user

Comment by Andrey Myatlyuk [ 06/May/13 ]

@Fedor
First of all, I am a user. We use Gradle at our company and we understand the drawbacks of using a product with a substantial development of features from version to version vs. 10-y.o. Maven and we just deal with it. We have our faith in Gradle team and willing to accept minor inconveniences over the benefits we are reaping. Also, I would like to remind you that Gradle is a free tool and it is open source software. Anyone can make changes there as they see fit.

Comment by Danno Ferrin [ 06/May/13 ]

I can say that this is one of the issues blocking my day job from switching any of our builds to Gradle. Peaceful coexistence with Maven is required.

Comment by Fedor Belov [ 06/May/13 ]

@Andrey

as you can see this may be important for someone. I can't fix it because I din't have enough time and knowledge. But I can vote for this issue and wait it to be fixed

Comment by Davide Cavestro [ 07/May/13 ]

It's too easy to end up with a maven cache containing only poms for some GAVs: if you have mavenLocal as an additional repo, then there's no hope for you

Seriously, I don't blame Gradle guys - they said it's on the list to get fixed - but simply give supporting evidence for this issue intricacy and importance.

Please note IDE integration is sometimes affected by this issue in an underhand manner: i.e. I have several subprojects using mavenLocal in order to support independent installs of internal libraries snapshots. This morning I spent more than an hour with GGTS simply ignoring all dependencies on jdom: there was a pom with no jars for jdom in my .m2 cache, just because some days ago I used maven to build a plugin.

Comment by Dave Syer [ 21/Aug/13 ]

This looks like an Ivy problem. See https://issues.apache.org/jira/browse/IVY-1432 for a workaround.

Comment by John Engelman [ 21/Aug/13 ]

So this particular issue is causing some issues with our organization because we have some Gradle and some Maven projects building on the same CI server...also locally for developers, it's a serious nightmare. I've dug through the code and I understand where the issue is, but I'm just looking for some direction from the Gradle guys on how they think it should get fixed.

Basically the issue is that in MavenResolver.java the isUsepoms() is used to check if the POM is present in the repository. If so, then it selects that repository to resolve the file. If MavenLocal() is your first repository, then it is selected and there is no mechanism for falling out of using that repository. I see 2 possible solutions:
1) Allow configuration of artifactUrls for the MavenLocal() repository.
2) Create a MavenLocalResolver that extends MavenResolver and overrides the getDependency method. Inside this method, after finding the POM file, parse its contents and check for the existence of the artifact itself.

Option 1 would be the simplest solution, though it wouldn't be very clean since most people would start adding a URL to Maven Central to the MavenLocal repository. It seems like we could handle more transparently to the user.
Option 2 would be the best because it basically ignores MavenLocal in these circumstances. It's just a more complicated solution.

I'm hoping to have a pull request together this week, so any thoughts would be awesome.

Comment by Mauro Molinari [ 22/Aug/13 ]

Please note the problem is not only with MavenLocal. In our use case, we have dependencies on libraries whose POM is on Maven Central, but the JAR is missing because of licensing issues. We want to provide that JAR through a local flat repository. However, this currently does not work, because if Gradle finds the POM on the Maven Central, it searches for the JAR on there and, since it doesn't find it, it makes the build fail.

Other similar use case: having the POM+JAR files on a repository, but the sources and/or Javadoc archives on a flat dir repository (because they are missing on the first one).

Comment by John Engelman [ 01/Sep/13 ]

@Mauro, I'm not quite sure where I'm understanding your issue. You mention a dependency where the POM is in Maven Central but the Jar isn't...You can resolve that issue by configuring an additionalUrl for the mavenCentral repository which accepts file URIs.

Comment by Mauro Molinari [ 02/Sep/13 ]

@John: I'm not sure I understood what you mean. Maybe you mean to use the artifactlUrls parameter of the mavenCentral DSL element. However, this would solve only part of my problem and would require us to set up a full Maven repository just to host JARs (all of them?).
Actually, my use case is the following: my script declares Maven Central and a second flatDir repository, in which we put all the binary JARs, source JARs or Javadoc JARs which are not supplied by the first Maven repository. Apart from our specific use case (MavenCentrl + flatDir), It would be quite natural that if Gradle can't find some piece of information on the first repository (whatever it is) it were looking on the second one (whatever it is). At least, it would be extremely useful if such a behaviour could be optionally enabled.

There are many cases:

  1. POM in the first repository, binary[+source+Javadoc] JAR[s] in the second one
  2. POM+binary JAR in the first repository, source JAR in the second one
  3. POM+binary JAR int he first repository, Javadoc JAR in the second one
  4. other weird kind of combinations (like POM+source JAR in the first repository, binary JAR in the second one, if this does make any sense at all, etc.)

If I understand the Gradle documentation properly, the artifaclUrls parameter of maven/mavenCentrl would help to solve just the first case.

Comment by Adam Murdoch [ 09/Sep/13 ]

This is fixed, but only for mavenLocal(). For all other repositories it is still considered an error for the artifact to be missing when the pom is present (unless packaging == 'pom')

Comment by Mauro Molinari [ 10/Sep/13 ]

So should we better open new issues for the other use cases? It is not possible anymore to open them from here, while I already opened a thread in the forums at http://forums.gradle.org/gradle/topics/making_gradle_find_a_jar_in_a_flatdir_repo_when_it_is_missing_from_a_maven_repo for my use case.

Comment by Adam Murdoch [ 10/Sep/13 ]

@Mauro, yes please.

Comment by Osian Hughes [ 11/Sep/13 ]

I see this was fixed yesterday, did it make it into the RC build released yesterday? if not, would it be possible to get it into RC2? Because another 3rd party framework we are using is now starting to use the local maven repo, and it is leaving this "pom only" structure all over the place in the local repo, so this fix would be very much appreciated.

Comment by Mauro Molinari [ 12/Sep/13 ]

@Adam: but how to do? I added a comment at http://forums.gradle.org/gradle/topics/making_gradle_find_a_jar_in_a_flatdir_repo_when_it_is_missing_from_a_maven_repo requesting the opening of a new issue. Or should I do in a different way?

Comment by Adam Murdoch [ 15/Sep/13 ]

@Mauro, that's fine.

Generated at Wed Jun 30 12:10:59 CDT 2021 using Jira 8.4.2#804003-sha1:d21414fc212e3af190e92c2d2ac41299b89402cf.