I have some more information on this. It appears to be specific to testng versions starting with 6.2

If I use 6.1.1, or 5.14.10, then my test task will correctly fail if there are exceptions thrown in any of methods annotated with the testng BEFORE annotations (BeforeClass, BeforeMethod, BeforeTest). However, they will not if using version 6.2/6.2.1.

Something changed in testng version 6.2 that IS NOT backwards compatible with however gradle is integrated with it for determining if there were configuration failures which caused the tests not to execute (and therefore be marked as SKIPPED).

Here are the release notes for testng 6.2: https://github.com/cbeust/testng/blob/master/CHANGES.txt

The only thing that looks suspicious there is this line:

Added: IConfigurationListener is now a public listener, along with a new one: IConfigurationListener2

Looking through the gradle source code as best as I can understand it, I think it uses IConfigurationListener to capture these kinds of configuration errors. Perhaps something about making this a public listener has messed things up.

The good news for me is that I'm unblocked, as I can just use version 6.1.1 and it works for me.

Let me know if you think it's a regression on TestNG.

I found a small amount of time to revisit and troubleshoot this issue. I have no experience with the gradle git codebase, troubleshooting or patching it, but I took a small stab. I'm fairly confident the issue is around the change that was made in Testng 6.2 to make the IConfigurationListener public.

Gradle's testng framework implements the internal hidden IConfigurationListener interface and translates any calls to onConfigurationFailure to errors as seen here:

public void onConfigurationFailure(ITestResult testResult)

[NOTE the System.out.println was from me working to troubleshoot the issue. I also had to modify gradle's CaptureTestOutputTestResultProcessor.StdOutForward class's onOutput method to log stdout messages in order to see the above stdout message]

I have a sample gradle project with one testng test class. It has a @BeforeClass annotated method that throws an exception.

When I run this build with debug logging using my customized gradle distribution, with testng 6.1.1, I see the following in the log file corresponding to configuration failure:

21:51:26.207 [QUIET] [system.out] 21:51:26.206 [INFO] [org.gradle.api.internal.tasks.testing.processors.CaptureTestOutputTestResultProcessor] stdOut: onConfigurationFailure
21:51:26.210 [QUIET] [system.out]
21:51:26.218 [DEBUG] [org.gradle.messaging.remote.internal.WorkerProtocol] Dispatching request to worker: [Request consumer: ecfc7c15-e960-4a82-999f-175299902bbf, payload: [MethodInvocation method: output()]

When I do the same with testng 6,2 I do not get that message in the log file.

So essentially, gradle is not getting notified of the configuration failure, and therefore has no clue there was a problem, so it doesn't fail the build.

This does indeed seem to be a regression on testng, though I'm not sure what the expected agreement was to suppose IConfigurationListener before it went public.

While normally I could live with just using 6.1.1, as it turns out I'm coming on board to using jmockit, and it looks like the magic sweet spot is to use jmocket-0.999-11 with testng 6.2. However, I can't use testng if its not going to tell me when there are configuration problems when executing my tests.

It seem there are other problems with gradle/testng integration as well, as I see now people are reporting that the recently released 6.3 of testng is not working with gradle.

While I would be more than happy to attempt to patch this, I think that's beyond my skills here, as I'm not sure how to patch gradle to support two different versions of the IConfigurationListener interface..

Let me know if I can help more troubleshooting this. Thanks.

Thanks for all the information. Very helpful!

Hi Luke, this problem still exists can you please reopen this bug. Tested on Gradle 1.12 and TestNG 6.8.7