[GRADLE-1710] TestNG test creation errors fail without test report Created: 02/Aug/11  Updated: 19/Nov/13  Resolved: 19/Nov/13

Status: Resolved
Project: Gradle
Affects Version/s: 1.0-milestone-3
Fix Version/s: 1.10-rc-1

Type: Bug
Reporter: Szczepan Faber Assignee: Szczepan Faber
Resolution: Fixed Votes: 1


 Description   

Reported initially at zendesk: http://support.gradleware.com/tickets/1028

When TestNG fails because it cannot instantiate a test class, gradle throws a stack (yeah!) but does not generate report (bummer). Gradle knows there is a test failure, and prints out the path to the reports directory, but that directory empty. In some pathological instances, the name of the problem test is not in the stack, which makes debugging difficult.

//TestNG 6.0.1 
@Test 
public class TryMeTestConstruct { 
  public class TryMeTestConstruct() { 
    // more realistically, this would be from member variable initialization 
    // throwing an exception: private String name = Constants.get("H"); 
    throw new NullPointerException(); 
  }

  public void gradleHasABug() { 
    Assert.assertTrue(true); 
  } 
}


 Comments   
Comment by Szczepan Faber [ 08/Aug/11 ]

According to the TestNg philosophy such situation is not a test failure but a 'configuration error' which should block the tests from running. That's the expected behavior, see this debate: http://groups.google.com/group/testng-users/browse_thread/thread/5206d4f82ff68549

The gradle behavior is the result of how TestNG works. In theory Gradle could work around and prepare a html result of such failure but that would require hacking because TestNG API does not support it. Also, it would be inconsistent with how TestNG rolls.

I suggest to use TestNG's @Before and @BeforeClass instead of constructors for all test-setup operations. Alternatively you can use jUnit instead of TestNG

Comment by Adam Murdoch [ 08/Aug/11 ]

I don't think we should close this. There's no reason we can't include this information in a report, so that you know what went wrong and where.

Comment by Szczepan Faber [ 09/Aug/11 ]

Could be.

It feels a bit of a waste to html-ize a single stack trace (regardless of the number of tests) that is already shown anyway (regardless of the gradle 'log' level).

I admit the build failure message is on the confusing side because it says to look for reports

Given all above I'm officially reopening the issue but I'm reducing the priority a bit.

Comment by Szczepan Faber [ 05/Oct/12 ]

At the very least, we should fix Gradle so that it does not tell the user to look for the report that does not exist. This issue popped up at the forums recently (the scenario with options.listeners configured with classes that do not exist on the test runtime classpath).

Comment by Celer Technologies [ 15/Apr/13 ]

Agree this is not good as the build fails no stack trace and then says look at the report where there is nothing.

Comment by Szczepan Faber [ 19/Nov/13 ]

The reports now include the stack trace of such failure.

Generated at Wed Jun 30 12:02:24 CDT 2021 using Jira 8.4.2#804003-sha1:d21414fc212e3af190e92c2d2ac41299b89402cf.