[GRADLE-2044] Cannot manually remove dependencies from milestone-8 cache structure Created: 12/Jan/12  Updated: 04/Jan/13  Resolved: 09/Feb/12

Status: Resolved
Project: Gradle
Affects Version/s: 1.0-milestone-4
Fix Version/s: 1.0-milestone-9

Type: Bug
Reporter: Chris Beams Assignee: Daz DeBoer
Resolution: Fixed Votes: 0

Attachments: Zip Archive GRADLE-2044.zip    

 Description   

Perhaps not strictly a bug, but entering it as such because this represents a kind of regression where users have always been able to do something like the following:

find ~/.gradle -name bundlor-plugin | xargs rm -rf

and expect it to work.

Against 1.0-milestone-8 (gradle-1.0-milestone-8-20120112000036+0100), once a dependency is downloaded, I cannot remove it from the cache.

I've attached a very simple project that reproduces this. Try it for yourself and you should see:

$ ./repro.sh 
+ rm -rf /tmp/testcache
+ ./gradlew --no-daemon --gradle-user-home /tmp/testcache hello
Download https://repo.springsource.org/plugins-snapshot/org/springframework/build/gradle/bundlor-plugin/0.1.1/bundlor-plugin-0.1.1.pom
Download https://repo.springsource.org/plugins-snapshot/org/springframework/build/gradle/bundlor-plugin/0.1.1/bundlor-plugin-0.1.1.jar
hello
:hello UP-TO-DATE

BUILD SUCCESSFUL

Total time: 4.502 secs
+ find /tmp/testcache -name bundlor-plugin
+ xargs rm -rfv
/tmp/testcache/caches/artifacts-8/filestore/org.springframework.build.gradle/bundlor-plugin/0.1.1/jar/a5177848dc819699d4f61258ab886db395d832f0/bundlor-plugin-0.1.1.jar
/tmp/testcache/caches/artifacts-8/filestore/org.springframework.build.gradle/bundlor-plugin/0.1.1/jar/a5177848dc819699d4f61258ab886db395d832f0
/tmp/testcache/caches/artifacts-8/filestore/org.springframework.build.gradle/bundlor-plugin/0.1.1/jar
/tmp/testcache/caches/artifacts-8/filestore/org.springframework.build.gradle/bundlor-plugin/0.1.1/pom/b629624774fff79dc7a1bbf1a4f391c677104cc8/bundlor-plugin-0.1.1.pom
/tmp/testcache/caches/artifacts-8/filestore/org.springframework.build.gradle/bundlor-plugin/0.1.1/pom/b629624774fff79dc7a1bbf1a4f391c677104cc8
/tmp/testcache/caches/artifacts-8/filestore/org.springframework.build.gradle/bundlor-plugin/0.1.1/pom
/tmp/testcache/caches/artifacts-8/filestore/org.springframework.build.gradle/bundlor-plugin/0.1.1
/tmp/testcache/caches/artifacts-8/filestore/org.springframework.build.gradle/bundlor-plugin
/tmp/testcache/caches/artifacts-8/module-metadata/org.springframework.build.gradle/bundlor-plugin/0.1.1/8898713ab8cfbd5fb61f54cf5cfb8d7d.ivy.xml
/tmp/testcache/caches/artifacts-8/module-metadata/org.springframework.build.gradle/bundlor-plugin/0.1.1
/tmp/testcache/caches/artifacts-8/module-metadata/org.springframework.build.gradle/bundlor-plugin
+ ./gradlew --no-daemon --gradle-user-home /tmp/testcache hello

FAILURE: Build failed with an exception.

* What went wrong:
Could not resolve group:org.springframework.build.gradle, module:bundlor-plugin, version:0.1.1.
Required by:
    :repro:unspecified
Cause: java.text.ParseException: /private/tmp/testcache/caches/artifacts-8/module-metadata/org.springframework.build.gradle/bundlor-plugin/0.1.1/8898713ab8cfbd5fb61f54cf5cfb8d7d.ivy.xml (No such file or directory) in file:/private/tmp/testcache/caches/artifacts-8/module-metadata/org.springframework.build.gradle/bundlor-plugin/0.1.1/8898713ab8cfbd5fb61f54cf5cfb8d7d.ivy.xml

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 1.551 secs

I consider this an important problem - no dependency management system is perfect, and it's critical that users can get under the hood and tinker with things to force a re-download, etc. I can already imagine folks criticizing this implementation as too complex, comparing it to the windows registry, etc. And that would be a shame, because in reality these changes are making Gradle better and better.

Thanks.



 Comments   
Comment by Daz DeBoer [ 07/Feb/12 ]

Interesting. The Gradle cache is tolerant to files being removed from /tmp/testcache/caches/artifacts-8/filestore/. The problem is that the "module-metadata" directory is linked from the binary file module-metadata.bin, which does not tolerate missing files.

The new --refresh dependencies switch should reduce your need to tinker with the cache. But making the cache tolerant to files being user-purged is certainly a good idea.

Comment by Chris Beams [ 08/Feb/12 ]

But making the cache tolerant to files being user-purged is certainly a good idea.

Frankly, I would consider this more like a "critical requirement" than a good idea. The ability to reason about what's in the cache and selectively and intuitively purge things from it is something that Maven and Ivy users do all the time. I am very impressed with the dependency resolution and cache improvements that have come along in recent milestones, but in my opinion these cannot come at the cost of turning the cache into a black box.

I don't think it's a far-fetched prediction that this issue, if unaddressed, will become a major source of complaints from users as Gradle becomes more and more popular. As it is right now, if the cache ends up with a dependency I do not want there, I have to wipe it out the cache completely, forcing an otherwise totally unnecessary re-downloading of all dependencies. I'm not sure what the semantics of `--refresh dependencies` are, and I hope it helps, but I can't encourage strongly enough that Gradle either (a) make it straightforward to search for and delete cached artifacts (i.e. `find ~/.gradle -name some-dep | xargs rm -rf`) or (b) provide a very friendly mechanism (preferably interactive) for cache management via `gradle` itself. For simplicity's sake, and in order to take advantage of all the goodness of the usual unix tools, option (a) is strongly preferred.

Thanks for the follow-up in any case.

Comment by Daz DeBoer [ 08/Feb/12 ]

Thanks for you encouragement and feedback.

One thing that sets the Gradle dependency cache apart from that in Maven or Ivy is that we utilise binary storage for much of our cache meta-data. This choice has made it easy for us to be much more sophisticated in our caching logic, and will allow us to easily extend this in the future. However, by using a binary format the cache has become necessarily opaque in some regards.

As an example, we currently cache the resolution of a dynamic version (1.+) to a static version (1.3) on a per-repository basis. This is stored in dynamic-versions.bin, and so cannot be manipulated by simple filesystem operations. In a similar way we cache the absence of a module in a repository, in module-metadata.bin.

I believe that the new '--refresh dependencies' switch will cater for most of the use cases that have traditionally required manual cache hackery. This switch functions almost the same as deleting the entire cache, except that artifacts previously downloaded are used as candidates to prevent re-download based on SHA1 key. So any binary cache data is ignored for resolution, but will be updated based on what's actually resolved.

Milestone 8 includes an initial chapter describing the dependency cache: http://gradle.org/docs/release-candidate/userguide/dependency_management.html#sec:dependency_cache. Hopefully that will clarify the way the cache currently works, even if it doesn't do anything to alleviate your concerns .

To the problem at hand: I consider it a bug that Gradle doesn't tolerate the remove of *.ivy files under the /module-metadata directory; this is an easy fix. However this won't provide a full cache management capability via the filesystem. It won't be possible to manually remove the cached absense of a module, or the cached resolution of a dynamic version. The fact that '--refresh dependencies' provides a better alternative will hopefully make that less of an issue.

Comment by Chris Beams [ 13/Feb/12 ]

Just to be clear - I see that this has been resolved for milestone-9. Does that mean that `--refresh dependencies` is being considered as the fix?

Comment by Daz DeBoer [ 13/Feb/12 ]

No. We've fixed the cache to be tolerant of manually removed artifacts. The following will no longer break Gradle:

find ~/.gradle -name bundlor-plugin | xargs rm -rf

However, there are some cache operations that you cannot easily do via manipulation of the filesystem. Examples include:

  • forcing Gradle to look for a module in a repository where it was previously found to be missing
  • forcing Gradle to re-evaluate a dynamic version like '1.+'

The only way to perform these via the filesystem would be to remove *.bin in the artifacts-N directory. So you'd still have any downloaded artifacts, but the calculated metadata would be discarded.

Comment by Chris Beams [ 14/Feb/12 ]

Great!

Comment by Chris Beams [ 23/Feb/12 ]

Just to check, the fixes mentioned above are milestone-9 only? I just tried the following against m8a with no luck as you can see:

cbeams@anakata:/Work/model-project-gradle[master]
$ find ~/.gradle -name docbook-reference-plugin | grep 8
/Users/cbeams/.gradle/caches/artifacts-8/filestore/org.springframework.build.gradle/docbook-reference-plugin
/Users/cbeams/.gradle/caches/artifacts-8/module-metadata/org.springframework.build.gradle/docbook-reference-plugin
cbeams@anakata:/Work/model-project-gradle[master]
$ find ~/.gradle -name docbook-reference-plugin | grep 8 | xargs rm -rf
cbeams@anakata:/Work/model-project-gradle[master]
$ gradle clean reference 
Note: the Gradle build daemon is an experimental feature.
As such, you may experience unexpected build failures. You may need to occasionally stop the daemon.

FAILURE: Build failed with an exception.

* What went wrong:
Could not resolve all dependencies for configuration ':classpath'.
> Could not resolve group:org.springframework.build.gradle, module:docbook-reference-plugin, version:0.1.2.
  Required by:
      :model-project:1.0.0.BUILD-SNAPSHOT
   > java.text.ParseException: /Users/cbeams/.gradle/caches/artifacts-8/module-metadata/org.springframework.build.gradle/docbook-reference-plugin/0.1.2/efdc0d20916eefdddb3920ec1099ea8.ivy.xml (No such file or directory) in file:/Users/cbeams/.gradle/caches/artifacts-8/module-metadata/org.springframework.build.gradle/docbook-reference-plugin/0.1.2/efdc0d20916eefdddb3920ec1099ea8.ivy.xml

Comment by Adam Murdoch [ 23/Feb/12 ]

@Chris, the fix is in m9.

Generated at Wed Jun 30 12:11:16 CDT 2021 using Jira 8.4.2#804003-sha1:d21414fc212e3af190e92c2d2ac41299b89402cf.