[GRADLE-1822] testng configuration failures resulting in skipped tests do not fail the build Created: 07/Oct/11  Updated: 17/Jun/14  Resolved: 04/Dec/11

Status: Resolved
Project: Gradle
Affects Version/s: 1.0-milestone-3
Fix Version/s: 1.0-milestone-7

Type: Bug
Reporter: Doug Lethin Assignee: Peter Niederwieser
Resolution: Fixed Votes: 0


 Description   

I have a TestNG test which is failing during the configuration phase. However, gradle ignores the failure and the test tasks succeeds. This is not the same behavior as maven which will fail the test. I took a look at the gradle source code as best I can and it looks like in the TestNgTestResultProcessorAdapter class, there are provisions in the onConfigurationFailure method to account for these failure and if I'm interpreting right create a fake test and mark it as failed so the the test task will end up failing. However, this does not appear to be working.

I'm tracking a post about this in the gradle forums -

http://forums.gradle.org/gradle/topics/is_there_a_way_to_configure_build_to_fail_on_skipped_tests_when_using_testng



 Comments   
Comment by Doug Lethin [ 10/Oct/11 ]

I have some more information on this. It appears to be specific to testng versions starting with 6.2

If I use 6.1.1, or 5.14.10, then my test task will correctly fail if there are exceptions thrown in any of methods annotated with the testng BEFORE annotations (BeforeClass, BeforeMethod, BeforeTest). However, they will not if using version 6.2/6.2.1.

Something changed in testng version 6.2 that IS NOT backwards compatible with however gradle is integrated with it for determining if there were configuration failures which caused the tests not to execute (and therefore be marked as SKIPPED).

Here are the release notes for testng 6.2: https://github.com/cbeust/testng/blob/master/CHANGES.txt

The only thing that looks suspicious there is this line:

Added: IConfigurationListener is now a public listener, along with a new one: IConfigurationListener2

Looking through the gradle source code as best as I can understand it, I think it uses IConfigurationListener to capture these kinds of configuration errors. Perhaps something about making this a public listener has messed things up.

The good news for me is that I'm unblocked, as I can just use version 6.1.1 and it works for me.

Comment by Cédric Beust [ 13/Oct/11 ]

Let me know if you think it's a regression on TestNG.

Comment by Doug Lethin [ 12/Nov/11 ]

I found a small amount of time to revisit and troubleshoot this issue. I have no experience with the gradle git codebase, troubleshooting or patching it, but I took a small stab. I'm fairly confident the issue is around the change that was made in Testng 6.2 to make the IConfigurationListener public.

Gradle's testng framework implements the internal hidden IConfigurationListener interface and translates any calls to onConfigurationFailure to errors as seen here:

public void onConfigurationFailure(ITestResult testResult)

{ System.out.println("onConfigurationFailure"); // Synthesise a test for the broken configuration method TestDescriptorInternal test = new DefaultTestMethodDescriptor(idGenerator.generateId(), testResult.getMethod().getTestClass().getName(), testResult.getMethod().getMethodName()); resultProcessor.started(test, new TestStartEvent(testResult.getStartMillis())); resultProcessor.failure(test.getId(), testResult.getThrowable()); resultProcessor.completed(test.getId(), new TestCompleteEvent(testResult.getEndMillis(), TestResult.ResultType.FAILURE)); }

[NOTE the System.out.println was from me working to troubleshoot the issue. I also had to modify gradle's CaptureTestOutputTestResultProcessor.StdOutForward class's onOutput method to log stdout messages in order to see the above stdout message]

I have a sample gradle project with one testng test class. It has a @BeforeClass annotated method that throws an exception.

When I run this build with debug logging using my customized gradle distribution, with testng 6.1.1, I see the following in the log file corresponding to configuration failure:

21:51:26.207 [QUIET] [system.out] 21:51:26.206 [INFO] [org.gradle.api.internal.tasks.testing.processors.CaptureTestOutputTestResultProcessor] stdOut: onConfigurationFailure
21:51:26.210 [QUIET] [system.out]
21:51:26.218 [DEBUG] [org.gradle.messaging.remote.internal.WorkerProtocol] Dispatching request to worker: [Request consumer: ecfc7c15-e960-4a82-999f-175299902bbf, payload: [MethodInvocation method: output()]

When I do the same with testng 6,2 I do not get that message in the log file.

So essentially, gradle is not getting notified of the configuration failure, and therefore has no clue there was a problem, so it doesn't fail the build.

This does indeed seem to be a regression on testng, though I'm not sure what the expected agreement was to suppose IConfigurationListener before it went public.

While normally I could live with just using 6.1.1, as it turns out I'm coming on board to using jmockit, and it looks like the magic sweet spot is to use jmocket-0.999-11 with testng 6.2. However, I can't use testng if its not going to tell me when there are configuration problems when executing my tests.

It seem there are other problems with gradle/testng integration as well, as I see now people are reporting that the recently released 6.3 of testng is not working with gradle.

While I would be more than happy to attempt to patch this, I think that's beyond my skills here, as I'm not sure how to patch gradle to support two different versions of the IConfigurationListener interface..

Let me know if I can help more troubleshooting this. Thanks.

Comment by Peter Niederwieser [ 04/Dec/11 ]

Thanks for all the information. Very helpful!

Comment by Donald Ngo [ 17/Jun/14 ]

Hi Luke, this problem still exists can you please reopen this bug. Tested on Gradle 1.12 and TestNG 6.8.7

Generated at Wed Jun 30 12:05:17 CDT 2021 using Jira 8.4.2#804003-sha1:d21414fc212e3af190e92c2d2ac41299b89402cf.