@Adam What is the current trunk behavior for this?

It hasn't changed (yet). You'll still get the duplicates.

We will improve this. The default should probably to throw an exception. You can specify excludes:

aproject.tasks.withType(Jar).each {archiveTask -> from zipTree( archiveTask.archivePath ).matching { exclude 'META-INF' }

Please see this issue I have with the potential behavior of a solution to this issue:
http://gradle.1045684.n5.nabble.com/Duplicate-files-in-archives-td3272391.html

In a nutshell, it'd be great to get some control over the merging of duplicates - in particular, being able to control which duplicate file gets into the final archive (as opposed to all copies of the file making it in, or the suggested solution of throwing an exception).

For me this would enable an "overrides" feature where configuration files can be dropped into a directory to override the default versions of these same files. The build would then put the overriden files into the archive.

For comparison, Ant's 'jar' task includes a 'duplicate' option with values "add", "preserve", and "fail" [1]. For some reason, "add" is the default, although "fail" would probably make more sense.

So using the Ant task may provide a workaround, if required. Some code to merge affected files can also be found in the 'jarWithDeps' task in [2].

[1] http://ant.apache.org/manual/Tasks/jar.html
[2] https://github.com/demobox/overthere-connection-checker/blob/master/build.gradle

This just cost me and a coworker about an hour of time. Please consider raising the priority.

I have done this:

jar {
	HashMap<String, Integer> listFiles = new HashMap<String, Integer>();
	union.each { col ->
		from(col) {
			eachFile { mFile ->						
				RelativePath relPath=mFile.getRelativePath();
				String rel=relPath.toString();
				if (!listFiles.containsKey(rel)) { 
					listFiles.put(rel, 1);
				} else {
					println("WARNING! File: " + rel + " is duplicated in " + relPath.getParent())
					mFile.exclude();
				}
			}
		}
	}
	.........
}

And it is working since it's excluding me a single duplicated file... But I'm still getting a jar with everything duplicated.

My union is a FileCollection and if I print it looks like:

D:\workspace\trunk\app\build\classes\main
D:\workspace\trunk\app\build\resources\main
D:\workspace\trunk\external\build\classes\main
D:\workspace\trunk\external\build\resources\main
D:\workspace\trunk\api\build\classes\main
D:\workspace\trunk\api\build\resources\main

If I go to "trunk" and I search for individual files, nothing is duplicated (not as in the jar file).

resources\main contains package structures with all the binary resources inside while "classes\main" contains packages structure with just .class

ok, this has something to do with this: http://forums.gradle.org/gradle/topics/gradle_doing_weird_things_duplicating_files_on_jar

jar {
    duplicatesStrategy = DuplicatesStrategy.EXCLUDE
}

see http://gradle.org/docs/current/javadoc/org/gradle/api/file/DuplicatesStrategy.html

It's a step in the right direction but there's something that should be considered (or maybe it already is being considered but I don't know about it ).
Folks might use this to avoid dependency issues, which I think is a bad idea. Problem here is that you don't know which version you wound up with after the fact unless you created a jar of jars instead of just classes. Suggest adding a new strategy:

DuplicatesStrategy.DEP_EXCLUDE

Proposed functionality is that it excludes all files from conflicting versions of libraries selecting files from the latest lib automagically. Of course there might be additional variations as it might be desirable to combine this with others. Alternatively you could force the user to use a dependency exclude by making it so that DuplicatesStrategy.EXCLUDE doesn't work on library classes. I don't care for the latter because it makes bringing over large projects from Maven a major pain. I'm doing exactly that right now.

Of course maybe this has already been thought of. If so, then the documentation could use an update to make the expected behavior clearer.

As announced on the Gradle blog we are planning to completely migrate issues from JIRA to GitHub.

We intend to prioritize issues that are actionable and impactful while working more closely with the community. Many of our JIRA issues are inactionable or irrelevant. We would like to request your help to ensure we can appropriately prioritize JIRA issues you’ve contributed to.

Please confirm that you still advocate for your JIRA issue before December 10th, 2016 by:

• Checking that your issues contain requisite context, impact, behaviors, and examples as described in our published guidelines.
• Leave a comment on the JIRA issue or open a new GitHub issue confirming that the above is complete.

We look forward to collaborating with you more closely on GitHub. Thank you for your contribution to Gradle!

@James I don't think we will want to convolute creating an archive with aspects of dependency management. Those should be kept separate. I think you can support your use case with the help of ArchiveCopyTask.duplicateStrategy and AbstractCopyTask.filter.