[GRADLE-2542] "Received unexpected module descriptor" exception if Ivy resolver returns different module than the one specified; e.g: foo#bar;default ==> foo#bar;1.2.3 Created: 31/Oct/12 Updated: 10/Feb/17 Resolved: 10/Feb/17
We would like to use a custom Ivy resolver that can turn symbolic names, e.g. "default" to actual library versions during the resolution process. This would allow us to manage library versions centrally for a multi-project build (allowing us to declare "default" versions only once in a multi project build and use "default" in every sub-project without having to specify the exact version in every single sub-project).
Example of how this would look like in a sub project: one could specify a dependency with default as the revision, like this:
The default version then would be turned to an actual version number by the resolver, returning a module description with the version number. This doesn't work as Gradle checks the returned descriptor against the expected module and fails, since "default" is not the same as say "1.2.3."; Gradle fails with error messages like this:
Gradle Ivy resolution should allow the resolver to return a different module version, thus allow the decision of which version of the module to pick up be deferred until the resolvers run.
|Comment by Peter Horvath [ 12/Nov/12 ]|
The exception is thrown from method org.gradle.api.internal.artifacts.ivyservice.ivyresolve.LazyDependencyToModuleResolver.StaticVersionResolveResult.onUnexpectedModuleRevisionId(ModuleDescriptor)
A possible solution would be to introduce a configuration option to allow using unexpected modules.
What do you think?
|Comment by Luke Daley [ 16/Nov/12 ]|
Why do you need to do this at the resolver level? Could you not do this by pre processing the dependencies before resolve?
|Comment by Peter Horvath [ 19/Nov/12 ]|
We are creating a central build configuration for multi-project builds. It will be imported by a number of separate builds within our firm using the apply directive.
We tried the pre-processing approach but since our Gradle build file is imported, we hit build life-cycle issues. It seems that Gradle doesn't support an imported build file to perform pre-processing on the build file it was imported from. The only possible solution we could find is deferring the dependency version look-up until the resolver is called and patch Gradle internally to prevent this exception from being thrown.
|Comment by Mathieu Gervais [ 19/Nov/12 ]|
The "default" conceptual release version is something we've been using very successfully in our current Ant based centralized build infrastructure. It allows us to define "stacks" of related releases that are created together.
Imagine an application depending on hundreds of dependencies.
We "bundle" up some of these into groups, to make the whole thing easier to manage. A bundle is thus a group of libraries versions that have been tested and known to work well together.
An application can thus take a dependency on such a bundle, and that will automatically define all the versions to use for these dependencies.
Note that if these dependencies were only transitive, the application would not have them at all in its list of dependencies, thus "default" would not be needed at all.
But the truth is that often, an application needs to take a direct dependency on some library that is also pulled in transitively. "default" is our way for this application to say "give me whatever version works well with my other dependencies", without the need for a human to manually pick the right version.
We'd be interested to hear if you have other suggestions on how to implement this "default" concept in gradle ivy resolution.
If not, we would very much like to see this issue fixed, maybe as suggestion by Peter above, so we can implement our own solution that we know works reasonably well.
By the way, since gradle has an enterprise focus, this might be a feature worth considering: we've found it dramatically helps manage large collections of dependencies that change at a rapid pace without imposing a tricky maintenance burden on all the application development teams.
|Comment by Peter Horvath [ 19/Nov/12 ]|
We've patched org.gradle.api.internal.artifacts.ivyservice.ivyresolve.LazyDependencyToModuleResolver.StaticVersionResolveResult.onUnexpectedModuleRevisionId(ModuleDescriptor): instead of throwing an exception we only log that the module descriptor received is different from the one expected. This approach seems to work well for us. Do you think a similar change could make its way to the official code base?
|Comment by Luke Daley [ 19/Nov/12 ]|
It's unclear at this point. I'll bring this issue to the attention of our guys who work the most in this area and have them assess the impact.
|Comment by Adam Murdoch [ 26/Nov/12 ]|
We do plan to bake the concept of "works well together" into dependency resolution, so that Gradle will prefer versions of things that are known to work well together over versions that "should work together". This would be driven by meta-data from a few sources:
It seems like we could solve your use case by providing some way for the consumer to specify some constraints, perhaps combined with some way to leave the version unspecified for a dependency declaration.
I would suggest not going with the resolver approach for a few reasons:
Instead, I would suggest we add some way to influence how the (group, module, version) specified for a dependency declaration is mapped to a set of candidate (group, module, version) identifiers. This would happen as part of dependency resolution, but before we attempt to search for these modules in any repository.
|Comment by Adam Murdoch [ 26/Nov/12 ]|
We'd need to make the change somewhat differently. The contract of ModuleVersionIdResolveResult.resolve() is that it returns the meta-data for the module specified by ModuleVersionIdResolveResult.getId(). The callers of this method rely on this contract. For example, DependencyGraphBuilder is going to apply conflict resolution using "default" as the version, rather than the actual version.
Instead, the responsibility of mapping "default" to a concrete version should live in an implementation of DependencyToModuleVersionIdResolver. The plan is to change LazyDependencyToModuleResolver.resolve() so that it always attempts to resolve the dependency, rather than deferring resolution of static versions. We'd also need to change UserResolverChange to defer the check whether a version is static or dynamic until after the dependency has been resolved.
This approach, however, has some backwards compatibility and performance implications. Currently, resolution will succeed if a static version that does not exist is later evicted by conflict resolution. The above change would cause this build to fail, which is, strictly speaking, a breaking change (but one we do want to make at some point). Related to this, this change would also mean that the module descriptor for evicted versions would be downloaded, even when not required due to being evicted. To deal with this, we intend to change the contract of DependencyToModuleVersionIdResolver.resolve() so that it instead returns a set of candidate versions, and DependencyGraphBuilder can later ask for the specific versions it needs as it resolves conflicts and traverses the dependency graph.
I think this works well with what you need to do, in that it sounds like you're interested in changing how the first resolution from requested version -> actual module version id works, but want the second resolution from module version id -> meta-data + artifacts to proceed normally.
|Comment by Mathieu Gervais [ 28/Nov/12 ]|
Thanks for the details. It's great to see that you see value in this type of feature. We are keen to work with you on moving this forward.
|Comment by Benjamin Muschko [ 15/Nov/16 ]|
As announced on the Gradle blog we are planning to completely migrate issues from JIRA to GitHub.
We intend to prioritize issues that are actionable and impactful while working more closely with the community. Many of our JIRA issues are inactionable or irrelevant. We would like to request your help to ensure we can appropriately prioritize JIRA issues you’ve contributed to.
Please confirm that you still advocate for your JIRA issue before December 10th, 2016 by:
We look forward to collaborating with you more closely on GitHub. Thank you for your contribution to Gradle!
|Comment by Benjamin Muschko [ 10/Feb/17 ]|
Thanks again for reporting this issue. We haven't heard back from you after our inquiry from November 15th. We are closing this issue now. Please create an issue on GitHub if you still feel passionate about getting it resolved.