[GRADLE-2672] Gradle reports contain incorrect values for test case times Created: 07/Feb/13  Updated: 16/Feb/13  Resolved: 16/Feb/13

Status: Resolved
Project: Gradle
Affects Version/s: None
Fix Version/s: None

Type: Bug
Reporter: Gradle Forums Assignee: Unassigned
Resolution: Not A Bug Votes: 0

Attachments: Zip Archive GRADLE-2672.zip    


I just upgraded to Gradle 1.4. I was excited to try out the new XML report output for CI as well as the new HTML report output for TestNG tests (big improvement!). But while testing it out I discovered that Gradle is reporting incorrect values for test case execution times when my tests are executed. I've tried running just the tests (gradle test) and with cobertura (gradle cobertura). Regardless, both the HTML files and the CI test output (TEST-<classname>) report execution times are all negative and of the incorrect magnitude. And on top of that the execution times differ for the same tests between each type of report.

Here are the times reported for three tests in build/test-results/TEST-com.xyz.MemoryAwareECStreamTemplateTest.xml:
<testsuite name="com.xyz.MemoryAwareECStreamTemplateTest" tests="7" failures="0" errors="0" timestamp="2013-02-06T01:19:52" hostname="myhost" time="0.0">
<testcase name="testShouldAddEstimatedObjectSizeToTotalMemoryBeforeInjection" classname="com.xyz.MemoryAwareECStreamTemplateTest" time="-1.360113592028E9"/>
<testcase name="testShouldBlockWhenTotalMemoryExceedsMaxMemory" classname="com.xyz.MemoryAwareECStreamTemplateTest" time="-1.360113592087E9"/>
<testcase name="testShouldMarkFutureWithFailureWhenExceptionOccurs" classname="com.xyz.MemoryAwareECStreamTemplateTest" time="-1.360113592412E9"/>

Here are snapshots of the HTML output for the same tests for comparison. Notice how the execution times of the same tests differ from the CI XML:

Oddly enough, the individual test case times reported in build/reports/tests/Gradle suite/Gradle test.xml appear to be correct. Though the total time seems high (Gradle reports the total build time of 29.975 secs on the command-line). Here's a snippet:
<testsuite name="Gradle test" failures="0" tests="263" time="41.872" errors="0">
<testcase name="testShouldAddEstimatedObjectSizeToTotalMemoryBeforeInjection" time="0.058" classname="com.xyz.MemoryAwareECStreamTemplateTest"/>
<testcase name="testShouldBlockWhenTotalMemoryExceedsMaxMemory" time="0.324" classname="com.xyz.MemoryAwareECStreamTemplateTest"/>
<testcase name="testShouldMarkFutureWithFailureWhenExceptionOccurs" time="0.0070" classname="com.xyz.MemoryAwareECStreamTemplateTest"/>

Environment info:

Gradle 1.4
Gradle build time: Monday, January 28, 2013 3:42:46 AM UTC
Groovy: 1.8.6
Ant: Apache Ant(TM) version 1.8.4 compiled on May 22 2012
Ivy: 2.2.0
JVM: 1.6.0_33 (Sun Microsystems Inc. 20.8-b03)
OS: Linux 3.1.10-gentoo-r1 i386

[1] https://d37wxxhohlp07s.cloudfront.net/s3_images/854304/gradle_negative_test_times1.png?1360162624
[2] https://d37wxxhohlp07s.cloudfront.net/s3_images/854305/gradle_negative_test_times2.png?1360162710

Comment by Derek Burket [ 12/Feb/13 ]

I think Gradle is off the hook for this issue. While building my reproducible sample I discovered that my project was pulling in an older version of TestNG (5.13.1). So I tried the latest version and it worked as expected. Further investigation revealed that the issue affects TestNG 5.13.x. Versions less than or greater than those work as expected. I've attached my sample in case it's useful in any way.

Comment by Adam Murdoch [ 16/Feb/13 ]

Thanks for letting us know.

Generated at Wed Jun 30 12:28:18 CDT 2021 using Jira 8.4.2#804003-sha1:d21414fc212e3af190e92c2d2ac41299b89402cf.