Uploaded image for project: 'Gradle'
  1. Gradle
  2. GRADLE-1050

Jar task should merge entries, not create duplicates, at least by default.

    Details

    • Type: Bug
    • Status: Resolved
    • Resolution: Fixed
    • Affects Version/s: 0.9
    • Fix Version/s: None

      Description

      For a mystery project, I preferred to take jars from a few upstream java projects and combine them into a single jar for convenient distribution. Following hints on the Gradle Wiki, I created a fourth java project and create a trivial build.gradle consisting essentially of this:

      jar {
      project.rootProject.subprojects.each( {aproject ->
      if ( project != aproject ) { // avoid recursion
      aproject.tasks.withType(Jar).each

      {archiveTask -> from zipTree( archiveTask.archivePath ) }

      }
      }
      }

      This almost works. It does not work in that the resulting jar has 3 or 4 manifests and multiple entries for some directories. Zip allows entries with duplicate names, little did I know.

      I suspect this is just an unintended consquence of the recent refactoring of Archive tasks. In any case, I expect, at a minimum, that there would be an easy setting to make "from zipTree" act more like the old Jar merge method. (I couldn't find anything promising in 30 minutes of api/code browsing.) What I really expect is that this would be the default behavior; it must be the 99% case for adding files to jars.

      Thanks for your consideration!

        Issue Links

          Activity

          Hide
          lqbweb Ruben added a comment - - edited

          I have done this:

          jar {
          	HashMap<String, Integer> listFiles = new HashMap<String, Integer>();
          	union.each { col ->
          		from(col) {
          			eachFile { mFile ->						
          				RelativePath relPath=mFile.getRelativePath();
          				String rel=relPath.toString();
          				if (!listFiles.containsKey(rel)) { 
          					listFiles.put(rel, 1);
          				} else {
          					println("WARNING! File: " + rel + " is duplicated in " + relPath.getParent())
          					mFile.exclude();
          				}
          			}
          		}
          	}
          	.........
          }
          

          And it is working since it's excluding me a single duplicated file... But I'm still getting a jar with everything duplicated.

          My union is a FileCollection and if I print it looks like:

          D:\workspace\trunk\app\build\classes\main
          D:\workspace\trunk\app\build\resources\main
          D:\workspace\trunk\external\build\classes\main
          D:\workspace\trunk\external\build\resources\main
          D:\workspace\trunk\api\build\classes\main
          D:\workspace\trunk\api\build\resources\main

          If I go to "trunk" and I search for individual files, nothing is duplicated (not as in the jar file).

          resources\main contains package structures with all the binary resources inside while "classes\main" contains packages structure with just .class

          ok, this has something to do with this: http://forums.gradle.org/gradle/topics/gradle_doing_weird_things_duplicating_files_on_jar

          Show
          lqbweb Ruben added a comment - - edited I have done this: jar { HashMap< String , Integer > listFiles = new HashMap< String , Integer >(); union.each { col -> from(col) { eachFile { mFile -> RelativePath relPath=mFile.getRelativePath(); String rel=relPath.toString(); if (!listFiles.containsKey(rel)) { listFiles.put(rel, 1); } else { println( "WARNING! File: " + rel + " is duplicated in " + relPath.getParent()) mFile.exclude(); } } } } ......... } And it is working since it's excluding me a single duplicated file... But I'm still getting a jar with everything duplicated. My union is a FileCollection and if I print it looks like: D:\workspace\trunk\app\build\classes\main D:\workspace\trunk\app\build\resources\main D:\workspace\trunk\external\build\classes\main D:\workspace\trunk\external\build\resources\main D:\workspace\trunk\api\build\classes\main D:\workspace\trunk\api\build\resources\main If I go to "trunk" and I search for individual files, nothing is duplicated (not as in the jar file). resources\main contains package structures with all the binary resources inside while "classes\main" contains packages structure with just .class ok, this has something to do with this: http://forums.gradle.org/gradle/topics/gradle_doing_weird_things_duplicating_files_on_jar
          Hide
          amitport Amit Portnoy added a comment -
          jar {
              duplicatesStrategy = DuplicatesStrategy.EXCLUDE
          }
          

          see http://gradle.org/docs/current/javadoc/org/gradle/api/file/DuplicatesStrategy.html

          Show
          amitport Amit Portnoy added a comment - jar { duplicatesStrategy = DuplicatesStrategy.EXCLUDE } see http://gradle.org/docs/current/javadoc/org/gradle/api/file/DuplicatesStrategy.html
          Hide
          jcllings James R. Collings added a comment - - edited

          It's a step in the right direction but there's something that should be considered (or maybe it already is being considered but I don't know about it ).
          Folks might use this to avoid dependency issues, which I think is a bad idea. Problem here is that you don't know which version you wound up with after the fact unless you created a jar of jars instead of just classes. Suggest adding a new strategy:

          DuplicatesStrategy.DEP_EXCLUDE

          Proposed functionality is that it excludes all files from conflicting versions of libraries selecting files from the latest lib automagically. Of course there might be additional variations as it might be desirable to combine this with others. Alternatively you could force the user to use a dependency exclude by making it so that DuplicatesStrategy.EXCLUDE doesn't work on library classes. I don't care for the latter because it makes bringing over large projects from Maven a major pain. I'm doing exactly that right now.

          Of course maybe this has already been thought of. If so, then the documentation could use an update to make the expected behavior clearer.

          Show
          jcllings James R. Collings added a comment - - edited It's a step in the right direction but there's something that should be considered (or maybe it already is being considered but I don't know about it ). Folks might use this to avoid dependency issues, which I think is a bad idea. Problem here is that you don't know which version you wound up with after the fact unless you created a jar of jars instead of just classes. Suggest adding a new strategy: DuplicatesStrategy.DEP_EXCLUDE Proposed functionality is that it excludes all files from conflicting versions of libraries selecting files from the latest lib automagically. Of course there might be additional variations as it might be desirable to combine this with others. Alternatively you could force the user to use a dependency exclude by making it so that DuplicatesStrategy.EXCLUDE doesn't work on library classes. I don't care for the latter because it makes bringing over large projects from Maven a major pain. I'm doing exactly that right now. Of course maybe this has already been thought of. If so, then the documentation could use an update to make the expected behavior clearer.
          Hide
          bmuschko Benjamin Muschko added a comment -

          As announced on the Gradle blog we are planning to completely migrate issues from JIRA to GitHub.

          We intend to prioritize issues that are actionable and impactful while working more closely with the community. Many of our JIRA issues are inactionable or irrelevant. We would like to request your help to ensure we can appropriately prioritize JIRA issues you’ve contributed to.

          Please confirm that you still advocate for your JIRA issue before December 10th, 2016 by:

          • Checking that your issues contain requisite context, impact, behaviors, and examples as described in our published guidelines.
          • Leave a comment on the JIRA issue or open a new GitHub issue confirming that the above is complete.

          We look forward to collaborating with you more closely on GitHub. Thank you for your contribution to Gradle!

          Show
          bmuschko Benjamin Muschko added a comment - As announced on the Gradle blog we are planning to completely migrate issues from JIRA to GitHub. We intend to prioritize issues that are actionable and impactful while working more closely with the community. Many of our JIRA issues are inactionable or irrelevant. We would like to request your help to ensure we can appropriately prioritize JIRA issues you’ve contributed to. Please confirm that you still advocate for your JIRA issue before December 10th, 2016 by: Checking that your issues contain requisite context, impact, behaviors, and examples as described in our published guidelines . Leave a comment on the JIRA issue or open a new GitHub issue confirming that the above is complete. We look forward to collaborating with you more closely on GitHub. Thank you for your contribution to Gradle!
          Hide
          bmuschko Benjamin Muschko added a comment -

          @James I don't think we will want to convolute creating an archive with aspects of dependency management. Those should be kept separate. I think you can support your use case with the help of ArchiveCopyTask.duplicateStrategy and AbstractCopyTask.filter.

          Show
          bmuschko Benjamin Muschko added a comment - @James I don't think we will want to convolute creating an archive with aspects of dependency management. Those should be kept separate. I think you can support your use case with the help of ArchiveCopyTask.duplicateStrategy and AbstractCopyTask.filter .

            People

            • Assignee:
              Unassigned
              Reporter:
              jroth John Roth
            • Votes:
              31 Vote for this issue
              Watchers:
              26 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development