[GRADLE-2964] When Gradle makes very large JAR files, they're corrupt Created: 26/Nov/13  Updated: 03/Mar/14  Resolved: 03/Mar/14

Status: Resolved
Project: Gradle
Affects Version/s: None
Fix Version/s: 1.12-rc-1

Type: Bug
Reporter: Gradle Forums Assignee: Luke Daley
Resolution: Fixed Votes: 0


 Description   

This build script generates a load of dummy resources, packs them into a jar, and verifies the jar using the command-line zip tool:

apply plugin: 'java'

def generatedResourcesDir = new File(project.buildDir, "generated-resources")

task generateResources << {
def rnd = new Random()
def buf = new byte[128 * 1024];
for (x in 0..250) {
def dir = new File(generatedResourcesDir, x.toString())
dir.mkdirs()
for (y in 0..250) {
def file = new File(dir, y.toString())
rnd.nextBytes(buf)
file.bytes = buf
}
}
}

sourceSets {
main {
output.dir(generatedResourcesDir, builtBy: generateResources)
}
}

task verifyJar(dependsOn: jar) << {
def command = 'zip -T ' + jar.archivePath
def process = command.execute()
def status = process.waitFor()
assert status == 0, "command ${command} failed:
apply plugin: 'java'

def generatedResourcesDir = new File(project.buildDir, "generated-resources")

task generateResources << {
def rnd = new Random()
def buf = new byte[128 * 1024];
for (x in 0..250) {
def dir = new File(generatedResourcesDir, x.toString())
dir.mkdirs()
for (y in 0..250) {
def file = new File(dir, y.toString())
rnd.nextBytes(buf)
file.bytes = buf
}
}
}

sourceSets {
main {
output.dir(generatedResourcesDir, builtBy: generateResources)
}
}

task verifyJar(dependsOn: jar) << {
def command = 'zip -T ' + jar.archivePath
def process = command.execute()
def status = process.waitFor()
assert status == 0, "command ${command} failed: \"${process.text.trim()}\""
}
check.dependsOn verifyJar
quot;${process.text.trim()}
apply plugin: 'java'

def generatedResourcesDir = new File(project.buildDir, "generated-resources")

task generateResources << {
def rnd = new Random()
def buf = new byte[128 * 1024];
for (x in 0..250) {
def dir = new File(generatedResourcesDir, x.toString())
dir.mkdirs()
for (y in 0..250) {
def file = new File(dir, y.toString())
rnd.nextBytes(buf)
file.bytes = buf
}
}
}

sourceSets {
main {
output.dir(generatedResourcesDir, builtBy: generateResources)
}
}

task verifyJar(dependsOn: jar) << {
def command = 'zip -T ' + jar.archivePath
def process = command.execute()
def status = process.waitFor()
assert status == 0, "command ${command} failed: \"${process.text.trim()}\""
}
check.dependsOn verifyJar
quot;"
}
check.dependsOn verifyJar

When i run this with gradle 1.9, it fails with:

FAILURE: Build failed with an exception.

  • Where:
    Build file '/home/tanderson/workspace/GradleZipBug/build.gradle' line: 37
  • What went wrong:
    Execution failed for task ':verifyJar'.
    > java.lang.AssertionError: command zip T /home/tanderson/workspace/GradleZipBug/build/libs/GradleZipBug.jar failed: "zip warning: missing end signature-probably not a zip file (did you
    zip warning: remember to use binary mode when you transferred it?)
    zip warning: (if you are trying to read a damaged archive try -F)

zip error: Zip file structure invalid (/home/tanderson/workspace/GradleZipBug/build/libs/GradleZipBug.jar)". Expression: (status == 0). Values: status = 3

I get the same error with 1.7, and a slightly different one with 1.8.

This is a pretty stupid build, but it's a distillation of a genuine case i have where i am building an uberjar containing an application and all its dependencies that ends up being absolutely gigantic - about 90 MB. That is also a stupid build, but it's the build i am currently chained to, and it's not likely to get substantially less stupid any time soon. However, with that build, it works fine under 1.7, and fails under 1.8 ( i have yet to try it with 1.9). When i examine the jar files produced by that build using 1.7 and 1.8, i see that the corrupt file produced by 1.8 has a truncated end of central directory record at the end; rather than being the 22-byte structure described in the specification, it is only 8 bytes, suggesting a buffering issue somewhere.

If you play with the parameters of the dummy resource generation to make more, smaller, files, you can get different results: 1.7 produces jars with valid central directories, whereas 1.8 does not. Sadly, this test still fails, because the command-line zip program cannot handle archives with more than 65535 entries, even though the various bits of jar support in Java can.

My Java is:

java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

My lsb_release -a is:

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 12.04.3 LTS
Release: 12.04
Codename: precise

Is this a known bug? Is there a workaround? Is this something to do with gradle, or a problem with the JDK?



 Comments   
Comment by Szczepan Faber [ 23/Jan/14 ]

This may be related to the limitations of the ZipOutputStream that we use internally for handling zips. Large zips (>4g, >65k entries) need extra handling (Zip64). Typically this is handled automatically by ZipOutputStream, but not always (see org.apache.tools.zip.ZipOutputStream#setUseZip64).

The example project produces about 62k files that weight 8g, hence it might be related. If my theory stands, a change in Gradle is needed to support those cases.

Generated at Wed Jun 30 12:36:11 CDT 2021 using Jira 8.4.2#804003-sha1:d21414fc212e3af190e92c2d2ac41299b89402cf.