[GRADLE-1050] Jar task should merge entries, not create duplicates, at least by default. Created: 24/Jul/10  Updated: 08/Feb/17  Resolved: 08/Feb/17

Status: Resolved
Project: Gradle
Affects Version/s: 0.9
Fix Version/s: None

Type: Bug
Reporter: John Roth Assignee: Unassigned
Resolution: Fixed Votes: 31

Issue Links:
Duplicate
Duplicated by GRADLE-2171 Add an option to avoid duplicate entr... Resolved
Related
Related to GRADLE-2177 When a subproject classesDir have bee... Resolved

 Description   

For a mystery project, I preferred to take jars from a few upstream java projects and combine them into a single jar for convenient distribution. Following hints on the Gradle Wiki, I created a fourth java project and create a trivial build.gradle consisting essentially of this:

jar {
project.rootProject.subprojects.each( {aproject ->
if ( project != aproject ) { // avoid recursion
aproject.tasks.withType(Jar).each

{archiveTask -> from zipTree( archiveTask.archivePath ) }

}
}
}

This almost works. It does not work in that the resulting jar has 3 or 4 manifests and multiple entries for some directories. Zip allows entries with duplicate names, little did I know.

I suspect this is just an unintended consquence of the recent refactoring of Archive tasks. In any case, I expect, at a minimum, that there would be an easy setting to make "from zipTree" act more like the old Jar merge method. (I couldn't find anything promising in 30 minutes of api/code browsing.) What I really expect is that this would be the default behavior; it must be the 99% case for adding files to jars.

Thanks for your consideration!



 Comments   
Comment by Hans Dockter [ 27/Jul/10 ]

@Adam What is the current trunk behavior for this?

Comment by Adam Murdoch [ 27/Jul/10 ]

It hasn't changed (yet). You'll still get the duplicates.

Comment by Hans Dockter [ 28/Jul/10 ]

We will improve this. The default should probably to throw an exception. You can specify excludes:

aproject.tasks.withType(Jar).each {archiveTask -> from zipTree( archiveTask.archivePath ).matching { exclude 'META-INF' }
Comment by Chris Brookes [ 22/Nov/10 ]

Please see this issue I have with the potential behavior of a solution to this issue:
http://gradle.1045684.n5.nabble.com/Duplicate-files-in-archives-td3272391.html

In a nutshell, it'd be great to get some control over the merging of duplicates - in particular, being able to control which duplicate file gets into the final archive (as opposed to all copies of the file making it in, or the suggested solution of throwing an exception).

For me this would enable an "overrides" feature where configuration files can be dropped into a directory to override the default versions of these same files. The build would then put the overriden files into the archive.

Comment by Andrew Phillips [ 19/Jul/11 ]

For comparison, Ant's 'jar' task includes a 'duplicate' option with values "add", "preserve", and "fail" [1]. For some reason, "add" is the default, although "fail" would probably make more sense.

So using the Ant task may provide a workaround, if required. Some code to merge affected files can also be found in the 'jarWithDeps' task in [2].

[1] http://ant.apache.org/manual/Tasks/jar.html
[2] https://github.com/demobox/overthere-connection-checker/blob/master/build.gradle

Comment by Andrew Spina [ 02/Mar/12 ]

This just cost me and a coworker about an hour of time. Please consider raising the priority.

Comment by Ruben [ 30/Oct/12 ]

I have done this:

jar {
	HashMap<String, Integer> listFiles = new HashMap<String, Integer>();
	union.each { col ->
		from(col) {
			eachFile { mFile ->						
				RelativePath relPath=mFile.getRelativePath();
				String rel=relPath.toString();
				if (!listFiles.containsKey(rel)) { 
					listFiles.put(rel, 1);
				} else {
					println("WARNING! File: " + rel + " is duplicated in " + relPath.getParent())
					mFile.exclude();
				}
			}
		}
	}
	.........
}

And it is working since it's excluding me a single duplicated file... But I'm still getting a jar with everything duplicated.

My union is a FileCollection and if I print it looks like:

D:\workspace\trunk\app\build\classes\main
D:\workspace\trunk\app\build\resources\main
D:\workspace\trunk\external\build\classes\main
D:\workspace\trunk\external\build\resources\main
D:\workspace\trunk\api\build\classes\main
D:\workspace\trunk\api\build\resources\main

If I go to "trunk" and I search for individual files, nothing is duplicated (not as in the jar file).

resources\main contains package structures with all the binary resources inside while "classes\main" contains packages structure with just .class

ok, this has something to do with this: http://forums.gradle.org/gradle/topics/gradle_doing_weird_things_duplicating_files_on_jar

Comment by Amit Portnoy [ 01/Mar/15 ]
jar {
    duplicatesStrategy = DuplicatesStrategy.EXCLUDE
}

see http://gradle.org/docs/current/javadoc/org/gradle/api/file/DuplicatesStrategy.html

Comment by James R. Collings [ 04/Dec/15 ]

It's a step in the right direction but there's something that should be considered (or maybe it already is being considered but I don't know about it ).
Folks might use this to avoid dependency issues, which I think is a bad idea. Problem here is that you don't know which version you wound up with after the fact unless you created a jar of jars instead of just classes. Suggest adding a new strategy:

DuplicatesStrategy.DEP_EXCLUDE

Proposed functionality is that it excludes all files from conflicting versions of libraries selecting files from the latest lib automagically. Of course there might be additional variations as it might be desirable to combine this with others. Alternatively you could force the user to use a dependency exclude by making it so that DuplicatesStrategy.EXCLUDE doesn't work on library classes. I don't care for the latter because it makes bringing over large projects from Maven a major pain. I'm doing exactly that right now.

Of course maybe this has already been thought of. If so, then the documentation could use an update to make the expected behavior clearer.

Comment by Benjamin Muschko [ 15/Nov/16 ]

As announced on the Gradle blog we are planning to completely migrate issues from JIRA to GitHub.

We intend to prioritize issues that are actionable and impactful while working more closely with the community. Many of our JIRA issues are inactionable or irrelevant. We would like to request your help to ensure we can appropriately prioritize JIRA issues you’ve contributed to.

Please confirm that you still advocate for your JIRA issue before December 10th, 2016 by:

  • Checking that your issues contain requisite context, impact, behaviors, and examples as described in our published guidelines.
  • Leave a comment on the JIRA issue or open a new GitHub issue confirming that the above is complete.

We look forward to collaborating with you more closely on GitHub. Thank you for your contribution to Gradle!

Comment by Benjamin Muschko [ 08/Feb/17 ]

@James I don't think we will want to convolute creating an archive with aspects of dependency management. Those should be kept separate. I think you can support your use case with the help of ArchiveCopyTask.duplicateStrategy and AbstractCopyTask.filter.

Generated at Wed Jun 30 11:45:38 CDT 2021 using Jira 8.4.2#804003-sha1:d21414fc212e3af190e92c2d2ac41299b89402cf.