I looked into the Cucumber JUnit runner a while ago and found it to be incompatible with Gradle. The general problem is that JUnit is very loose on how a JUnit runner is supposed to behave, which opens up chances for incompatibility.

Peter, thanks for the reply, I think I have found several issues that this might answer. I have an issue with a DBUnit test running in Maven and NOT in Gradle.

But, so I can better understand this incompatibility, can you please explain a little more what this shortcoming is for my own knowledge?

Is this an issue with the Gradle JUnit task?

Also, do you think there is any work-around such as calling Ant or Maven to run the JUnits so I can continue to use Gradle?

> I think I have found several issues that this might answer. I have an issue with a DBUnit test running in Maven and NOT in Gradle.

That's likely a different matter. When tests succeed in Maven but not in Gradle, it typically means that they are inadvertently tied to their environment. For example, some of the tests might only work when executed in a particular order. Such problems often go undetected until tests are executed in a different environment. The solution is to identify the problematic tests and fix them. Sometimes test setup or cleanup needs to be improved. As a last resort, it can also help to run the problematic tests with a separate test task (and thus in a separate JVM).

> But, so I can better understand this incompatibility, can you please explain a little more what this shortcoming is for my own knowledge?

The Cucumber JUnit runner uses org.junit.runner.notification.RunNotifier in a way that Gradle isn't expecting. I have yet to see this problem with any other JUnit runner.

> Is this an issue with the Gradle JUnit task?

I can't say at this point. This could be a Cucumber bug, a Gradle bug, or an incompatibility between the two that one side will need to work around. Likely the Cucumber runner was tested with (and possibly tweaked for) Maven but not Gradle.

> Also, do you think there is any work-around such as calling Ant or Maven to run the JUnits so I can continue to use Gradle?

You could use the Ant JUnit task or call the Maven executable. Except in the Cucumber case, I'd recommend to first investigate the problem. Chances are that it's a problem with the tests, in which case I'd try to fix them.

> Cucumber JUnit runner uses org.junit.runner.notification.RunNotifier in a way that Gradle isn't expecting. I have yet to see this problem with any other JUnit runner.

can you be specific about what is unusual here?

Sorry, I don't remember at this point. Probably the RunNotifier contract wasn't honored in one way or another, like not calling testStarted() and testFinished() in pairs under every circumstances. But I'd have to debug Cucumber again to provide a definitive answer.

Actually, I think the problem was that fireTestStarted() and fireTestFinished() were being called in a nested fashion, whereas RunNotifier's Javadoc says that they are (only) meant to be called for "atomic tests". From what I remember, Gradle couldn't cope with the nesting.

ok thanks, i'll have a dig around.

There appears to be a gradle bug here as a result of the difference between a cuke scenario and a junit test.

JUnitTestEventAdapter#testStarted creates a DefaultTestDescriptor and uses this call to calculate the name of that test

// Use this instead of Description.getMethodName(), it is not available in JUnit <= 4.5
private String methodName(Description description) {
Matcher matcher = methodStringMatcher(description);
if (matcher.matches()) {
return matcher.group(1);
}
return null;
}

cucumber-jvm does not work this way, tests are really scenarios and steps that are described in gherkin feature files. For example, a feature file could be

Feature: Addition
In order to avoid silly mistakes
As a math idiot
I want to be told the sum of two numbers

Scenario: Add two numbers
Given I have entered 1 into the calculator
And I have entered 2 into the calculator
When I press add
Then the result should be 3 on the screen

Each scenario will be presented as a junit test and each step (the given/when/then lines) will also appear as a test.

The junit runner contains no methods, e.g.

mport cucumber.junit.Cucumber;
import org.junit.runner.RunWith;

@RunWith(Cucumber.class)
@Cucumber.Options(format=

, features = "classpath:", glue = "classpath:", tags = "@beta")
public class BetaCukeRunner {
}

Given this setup, `JUnitTestEventAdapter#methodName` will create a descriptor with name = null. The JUnit Description passed in looks valid as its name is the scenario name and it has children (the individual given/when/then steps).

Later on, the following code will be called in `JUnitXmlReportGenerator` (ultimately from `JUnitTestEventAdapter#testStarted`)

@Override
protected void started(TestState state) {
TestDescriptorInternal test = state.test;
// NPE here - test.getName() is null
if (test.getName().equals(test.getClassName())) {
testSuiteReport = documentBuilder.newDocument();
rootElement = testSuiteReport.createElement("testsuite");
testSuiteReport.appendChild(rootElement);
// Add an empty properties element for compatibility
rootElement.appendChild(testSuiteReport.createElement("properties"));
outputs.put(TestOutputEvent.Destination.StdOut, new StringBuilder());
outputs.put(TestOutputEvent.Destination.StdErr, new StringBuilder());
testSuite = state;
}
}

and hence you blow with an NPE. JUnit responds to the failure by removing the failed listener and continuing on its way.

There may still be a problem with how you interpret cucumber results as 1 scenario looks likely to result in the appearance of 3 "junit" tests but not sure if that is really an issue or not. It's not obvious from the code and I can't progress beyond the above without patching gradle.

is this considered to be a gradle bug (and hence might get fixed in future) or should I plan on the basis that gradle junit support is for traditional junit unit testing only? I'm not sure where to begin offering a fix as it seems somewhat fundamental to how gradle views junit and how test results reported etc (ie it looks like the sort of thing where a 1-2line fix breaks more things than it fixes)

I don't consider this a Gradle bug, as Gradle just adheres to the JUnit contract (and Cucumber doesn't). That said, maybe we can find a way to make Cucumber work too. However this will require some wider investigations. For example, I don't know if and how this affects Gradle's JUnit reports. It's not something that the Gradle team will likely have time for soon. Ideally an external contributor would drive things forward and discuss his findings with us.

when you say "the junit contract", what exactly are referring to? Tests that are described by the standard junit annotations?

I'd say the NPE is a bug (as the method may return null but you behave as if it doesn't), the cucumber-jvm issue is an incompatibility.

I'm referring to the Javadoc for RunNotifier, which clearly says that fireTestStarted/fireTestFinished should only be called for atomic tests (which implies that nesting of these calls is impossible). All other JUnit-compatible testing frameworks I've seen (more specifically their @RunWith runners) respect that contract. Now I do understand why Cucumber doesn't, but that still makes it an incompatibility from Cucumber's side.

My first guess is that this problem isn't too hard to solve. The bigger question is how supporting arbitrary nested tests plays with our JUnit reports, our listener mechanism, etc. To drive this forward in a timely manner, we'll need some help from outside.

I haven't checked in detail, but I think that the NPE is a consecutive error of Cucumber incorrectly calling fireTestStarted() for a non-atomic test. Might still make sense to make this more robust, though.

the NPE is because you expect something like

public class MyTests {
@Test public void testThis() {
}
}

so the junit test `Description` would have `className` = MyTests and `methodName` = testThis

what you actually get is a description where the class name is the "scenario name" and the "method name" is the step. In this setup, both values are plain english phrases so they don't match the regex and you get null back.

This causes your listener to be removed & so you get no notifications. I would not be surprised if things worked normally without this (and with an appropriate change in your xml reporter).

I will try and get time to make a custom build to test out the above.

I have confirmed the above NPE bug with a patch against the latest gradle from github, removing the NPE allows cucumber to executes the tests though the output (no of tests run, html reports) is a bit dubious.