I would say that this is critical. TestNG is vastly superior to JUnit, JUnit4 is a pale immitation of TestNG. I would have thought supporting TestNG rather than JUnit would have been a good idea.

If the roadmap is flexible (as stated on the user mailinglist), I too would like to have this moved to an earlier release than "somewhere before 1.0".

For me lack of support for TestNG is a complete blocker for any sort of use of Gradle on a Java project. With all the talk of hierarchy configuration, I would like to see test framework get a higher priority.

It is easy to use TestNg with Gradle via its Ant task. This is not as convenient as having it as part of our Java plugin but nobody is blocked from using TestNG when using the current version of Gradle.

So what is the official idiom to achieve this?

Hans, perhaps you could add an example template of what you would do to use TestNG instead of JUnit to guide the rest of us as to how to progress using TestNG rather than the inferior JUnit.

I'm trying to get this working myself, I've created an AntTestNG, TestNGOptions and an AbstractAntTestTask classes and made the current Test class use AbstractOptions and AbstractAntTestTask instead of JunitOptions and AntJunit and added a useTestNG method so you can do test

.

I'll let you now when I get around to testing it.

this is what I've created so far, haven't tested it yet

I started creating a Bazaar branch with these files in in order to try things out, but I realized it wasn't clear if there were any unit tests in these 4 files – they all seem to be files destined for src/main/groovy.

I will publish the branch I have to lp:~russel/gradle/testng momentarily.

As I said I haven't tested it yet (when I try to build the :testCompile task runs out of memory... - I'll set up an IntelliJ project so I can try to test it).

Compilation works for me, the problem is in the running of the tests. The Test class is causing some problems.

Hans is doing a Groovy -> Java refactoring and this has changed the Test class amongst others. I have amended my TestNG branch which is published as a Bazaar branch on Launchpad at lp:~russel/gradle/testng. Basically things have moved on such that the patches attached to this issue are not usable without changes.

I currently get a couple of test failures, I'll see if I can get them sorted.

If anyone has any ideas for new unit tests or amendments of existing unit tests for this stuff pass them to me and I can add them into the branch.

put some more time in testng support, standard testng report is saved into the build/report/test directory.

additional work required for:

• support most important testng ant task options.
• based on annotations option use the correct testng jar [JavaDoc=> testng-5.8-jdk14.jar, JDK=> testng-5.8-jk15.jar].
• use the junitreport task to generate a junit report based on the testng test results (without loosing package/class information = currently the case thats why I've commented out the code).
• ...

I have applied the bundle to my branch and hav e pushed it up. A rebase against the latest Subversion head is included so revisions identities will have changed. Running the Gradle tests I get 2 failures and 1 error.

I have also got a Git repository tracking the same changes in its testng branch. The repository is at git@github.com:russel/gradle.git

Having installed the new Gradle despite the test fails, then on the example project above I get:

> gradle test
Executing Task: :init
Executing Task: :resources
Executing Task: :compile
Executing Task: :testResources
Executing Task: :testCompile
Executing Task: :test
[ant:taskdef] Could not load definitions from resource testngtasks. It could not be found.

Build failed with an exception.
Run with -s or -d option to get more details. Run with -f option to get the full (very verbose) stacktrace.

Build file '/home/users/russel/Progs/OddsByLanguage/Gradle/TestNG/build.gradle'

Execution failed for task :test.
Cause: Problem: failed to create task or type testng
Cause: The name is undefined.
Action: Check the spelling.
Action: Check that any custom tasks/types have been declared.
Action: Check that any <presetdef>/<macrodef> declarations have taken place.

BUILD FAILED

Total time: 3.707 secs
>

Russel that's most likely caused by the the taskdef containing a absolute path to the testng.jar file, the testng.jars need to get moved into the lib dir of gradle and the correct one needs to be used depending on the annotation option.

Isn't there a way of using Gradle or Ivy to download this to the right place rather than have it manually placed in the right place?

I removed the classpath from the line to leave:

ant.taskdef([resource : "testngtasks"]) //, classpath : "/home/teyckmans/lib/org/gradle/checkout/testng/testng-tester/lib/testng-5.8.jar"])

and I added a symbolic link in my $GRADLE_HOME/lib to a TestNG 5.8 JDK1.5 jar and now I get:

> gradle test
Executing Task: :init
Executing Task: :resources
Executing Task: :compile
Executing Task: :testResources
Executing Task: :testCompile
Executing Task: :test
[TestNGAntTask] TESTNG PASSED @/tmp/testng23599 WHICH CONTAINS:
[TestNGAntTask] -d
[TestNGAntTask] /home/users/russel/Progs/OddsByLanguage/Gradle/TestNG/build/reports/tests
[TestNGAntTask] -annotations
[TestNGAntTask] JDK
[TestNGAntTask] -testclass
[TestNGAntTask] /home/users/russel/Progs/OddsByLanguage/Gradle/TestNG/build/test-classes/org/mediastuffx/gradle/testng/UserImplTest.class
[TestNGAntTask] -suitename
[TestNGAntTask] Ant suite
[TestNGAntTask] -testname
[TestNGAntTask] Ant test

Build failed with an exception.
Run with -s or -d option to get more details. Run with -f option to get the full (very verbose) stacktrace.

Build file '/home/users/russel/Progs/OddsByLanguage/Gradle/TestNG/build.gradle'

Execution failed for task :test.
Cause: There were failing tests. See the report at /home/users/russel/Progs/OddsByLanguage/Gradle/TestNG/build/reports/tests.

BUILD FAILED

Total time: 5.007 secs
>

I tried the new Gradle on my ADS library and:

> gradle test
Executing Task: :init
Executing Task: :resources
Executing Task: :compile
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Executing Task: :testResources
Executing Task: :testCompile
Executing Task: :test
[TestNGAntTask] TESTNG PASSED @/tmp/testng40492 WHICH CONTAINS:
[TestNGAntTask] -d
[TestNGAntTask] /home/users/russel/Repositories/Bazaar/Masters/ADS/build/reports/tests
[TestNGAntTask] -annotations
[TestNGAntTask] JDK
[TestNGAntTask] -testclass
[TestNGAntTask] /home/users/russel/Repositories/Bazaar/Masters/ADS/build/test-classes/org/devjavasoft/ads/tests/Array_Test.class
[TestNGAntTask] /home/users/russel/Repositories/Bazaar/Masters/ADS/build/test-classes/org/devjavasoft/ads/tests/Bag_Test.class
[TestNGAntTask] /home/users/russel/Repositories/Bazaar/Masters/ADS/build/test-classes/org/devjavasoft/ads/tests/ComparablePair_Test.class
[TestNGAntTask] /home/users/russel/Repositories/Bazaar/Masters/ADS/build/test-classes/org/devjavasoft/ads/tests/Comparator_Test.class
[TestNGAntTask] /home/users/russel/Repositories/Bazaar/Masters/ADS/build/test-classes/org/devjavasoft/ads/tests/ContainerIteration_Test.class
[TestNGAntTask] /home/users/russel/Repositories/Bazaar/Masters/ADS/build/test-classes/org/devjavasoft/ads/tests/DLList_Test.class
[TestNGAntTask] /home/users/russel/Repositories/Bazaar/Masters/ADS/build/test-classes/org/devjavasoft/ads/tests/Heap_Test.class
[TestNGAntTask] /home/users/russel/Repositories/Bazaar/Masters/ADS/build/test-classes/org/devjavasoft/ads/tests/Pair_Test.class
[TestNGAntTask] /home/users/russel/Repositories/Bazaar/Masters/ADS/build/test-classes/org/devjavasoft/ads/tests/PriorityQueue_Test.class
[TestNGAntTask] /home/users/russel/Repositories/Bazaar/Masters/ADS/build/test-classes/org/devjavasoft/ads/tests/Queue_Test.class
[TestNGAntTask] /home/users/russel/Repositories/Bazaar/Masters/ADS/build/test-classes/org/devjavasoft/ads/tests/SLList_Test.class
[TestNGAntTask] /home/users/russel/Repositories/Bazaar/Masters/ADS/build/test-classes/org/devjavasoft/ads/tests/Searching_Test.class
[TestNGAntTask] /home/users/russel/Repositories/Bazaar/Masters/ADS/build/test-classes/org/devjavasoft/ads/tests/Set_Test.class
[TestNGAntTask] /home/users/russel/Repositories/Bazaar/Masters/ADS/build/test-classes/org/devjavasoft/ads/tests/Sorting_Test.class
[TestNGAntTask] /home/users/russel/Repositories/Bazaar/Masters/ADS/build/test-classes/org/devjavasoft/ads/tests/Stack_Test.class
[TestNGAntTask] /home/users/russel/Repositories/Bazaar/Masters/ADS/build/test-classes/org/devjavasoft/ads/tests/TreeIterator_Test.class
[TestNGAntTask] /home/users/russel/Repositories/Bazaar/Masters/ADS/build/test-classes/org/devjavasoft/ads/tests/filesort/FileSort_Test.class
[TestNGAntTask] -suitename
[TestNGAntTask] Ant suite
[TestNGAntTask] -testname
[TestNGAntTask] Ant test

BUILD SUCCESSFUL

Total time: 17.404 secs
>

This raises the question whether a TestNG jar should be shipped with Gradle?

I think BXXXXX (censored does not, but pulls it if it is ever used, this seems like a good idea to me.

Would be great if it's pulled in when needed. Haven't tasted much off the Ivy/dependency API's but I'll give it a shot.

I was looking at this some more and noticed that the JavaPlugin / GroovyPlugin modified some of the test options, propably Junit specific options (TestNG tests are always run in a forked JVM).

I've made an attempt to make the test class and the java+groovy plugin classes more test framework independent. (hope you like it).
I've added a TestPolicy class to separate the test class initialization done in the java and groovy plugins and added. I've also added an init method to TestPolicy to initialize the options and antTestTask attributes on the Test class depending on the test framework that needs to be used. (JunitTestPolicy <> TestNGTestPolicy)
I've also made it so that there can be a configurable default TestPolicy (currently hard coded to Junit).

Second patch file fixes the test class tests

Will try and do some more work on getting more testng features in the options.

Added dependency on TestNG when needed.

Only problem left is to get a hold of the absolute path of the testng jar to perform the taskdef (currently it scans the classpath passed to the antTestNGTask.execute for a file that starts with 'testng' - not a very nice way).

now all you have to do to use testng is:

dependencies {
addMavenRepo()
}
test{
useTestNG()
}

next I'll try and implement some more of the TestNG features.

got the suite stuff running:

test {
	useTestNG()

    options.suiteXmlBuilder().suite(name: 'testing-testng') {
        test (name : 'testing-testng', annotations : 'JDK', verbose:'1') {
            classes {
                'class' (name: 'org.mediastuffx.gradle.testng.UserImplTest')
            }
        }
    }
}

This not very nice but this does allow you to do everything the testng suite files allow with groovy flexibility.

I'm thinking of adding some functions on the TestNGOptions that allow easy creation of suites. Like for all classes that match certain patterns and generate a suite that preserves package information in the test names.

I made the assumption that test-tests-pass.patch and testpolicy.patch are both supeceded by testng-dep-when-needed.patch and applied it to my rebased to the latest trunk branch. I am a little worried that some of the changes in the TestNG branch are actually reversions of things Hans and Adam have added to trunk.

I think we need to tidy up the patches, perhaps even resync?

I am getting 1 error in DefaultProjectTest.testTaskWithConfigureClosure:

No signature of method: org.gradle.api.Task$$EnhancerByCGLIB$$4cdcedcf.javaConfigure() is applicable for argument types: (org.gradle.api.internal.project.DefaultProjectTest$_testTaskWithConfigureClosure_closure29) values:

and 2 failures:

JunitOptionsTest.testOptionMapWithTrueFalseValues:

true: expected array was null

ProjectLoadingIntegrationTest.handlesSimilarlyNamedBuildFilesInSameDirectory

Expected: not null got: null

Russel I started from lp:~russel/gradle/testng each patch contains one commit so all patches need to be applied. I haven't checked out from anywhere else so no sure how things could of gotten mixed up.

Think your getting the because not all patches are applied.

Not sure what could cause the JunitOptionsTest to fail (perhaps it's already fixed - doesn't fail on my current codebase).

I've checked the ProjectLoadingIntegrationTest and that seems to be an unintended change (sorry).

Sorry for the trouble looked at the error and also my fault, DefaultProjectTest.testTaskWithConfigureClosure my blind faith in my editor seems misplaced, javaConfigure should be configure there.

I'll review my current changes tomorrow before I add another patch.

I have been rebasing the Bazaar branch so as to ensure all the changes from Subversion repository trunk are slipped in in front of all changes associated with TestNG support. I know this violates lots of people's views on how to use DVCS, but it is the sanest way of tracking changes for commit to the Subversion repository later. The important point is that the TestNG changes must incorporate all the changes made to trunk since the branch was taken. The upshot of this is that lp:~russel/gradle/testng changes in ways that need rebasing to.

I believe that the 2008-11-01 and 2008-11-02 patches actually have a lot of the same edits in so I assumed they were acting as replacements.

So I wonder if it is appropriate to resynchronize so as to ensure the TestNG changes are all in place and are harmonious with trunk as it is now?

Something very suspicious appears to be going on, a whole load of changes that I thought had previously been made are being made again. I wonder if there was a problem introduced during one of the rebasings.

I guess there is an argument for creating a new branch with the files Tom currently has that all work, put straight on to a new branch of trunk.

Tom, Perhaps you can make a tarball of your working directory and email it to me?

Russel I've put an archive of my working directory here: http://www.mediastuffx.org/gradle-testng.tar.bz2. Not sure if did something wrong. If you find out I'd like to know.

I downloaded the tarball and on opening it certainly compiles and tests without error.

I pulled the changes into a fresh branch of Subversion Trunk and there is one error. As far as I can tell from having tried to run the changes in different ways, the error is a real one and is to do with the interaction between the TestNG code and the changed made to Trunk since the orginal branch for the TestNG stuff was taken.

lp:~russel/gradle/testng represents the current state for me.

ProjectLoadingIntegrationTest	handlesSimilarlyNamedBuildFilesInSameDirectory	Failure	 Expected: not null got: null

junit.framework.AssertionFailedError:
Expected: not null
got: null

Expected: not null
got: null

at org.gradle.integtests.AbstractIntegrationTest.getTestBuildFile(AbstractIntegrationTest.java:64)
at org.gradle.integtests.ProjectLoadingIntegrationTest.handlesSimilarlyNamedBuildFilesInSameDirectory(ProjectLoadingIntegrationTest.java:27)
	0.277

Russel, The ProjectLoadingIntegrationTest fails because of an unintended change. The filename of the 'similarly-named-build.gradle' to 'similarly-named build.groovy' so the integration test just doesn't find the file anymore.

After the rename all tests pass again.

Splendid, thanks for hunting that one down. The required edit was to change 'similarly-name-build.gradle' to 'similarly-named build.gradle' which is the file name in the code of Trunk.

OK, now all the tests pass.

I will install this version of Gradle and see if my TestNG using projects build.

Well it seems to be working but I have to say that having to use:

options.suiteXmlBuilder ( ).suite ( name : 'testing-testng' ) {
  test ( name : 'testing-testng' , annotations : 'JDK' , verbose : '1' ) {
    classes {
      'class' ( name : 'org.mediastuffx.gradle.testng.UserImplTest' )
    }
  }
}

seems a bit clumsy. What I really want to be able to say is:

include ( 'org.mediastuffx.gradle.testng.UserImplTest' )

Another possibility is:

xmlSuite ( 'testngTestSuite.xml' )

for where the suite already exists – though I prefer to use includes myself.

I have also found that if I do "gradle clean test" then everything works fine every time. However if I do "gradle clean test" followed by "gradle test" I get:

|> gradle test
TestNGTestPolicy [init]
:init
:resources
:compile
:testResources
:testCompile
:test
[ant:testng] [ERROR]: No test suite found.  Nothing to run

Build failed with an exception.
Run with -s or -d option to get more details. Run with -f option to get the full (very verbose) stacktrace.

Build file '/home/users/russel/Progs/OddsByLanguage/Gradle/TestNG/build.gradle'

Execution failed for task :test.
Cause: There were failing tests. See the report at /home/users/russel/Progs/OddsByLanguage/Gradle/TestNG/build/reports/tests.

BUILD FAILED

Total time: 4.422 secs
|> 

I am thinking of deleting the patches on this issue in favour of pointing people at lp:~russel/gradle/testng for a Bazaar branch or git://github.com/russel/gradle.git for a Git repository with a testng branch.

Russel, I agree completely that the suiteXmlBuilder is a very clumsy indeed. I don't think many people require that level of flexibility.

Include should not be to hard to add with the nested element 'classfileset' of the testng task. Not sure if it is possible to use it in combination with suites. I'll have a look at it.

Somthing like XmlSuite is already available through the TestNGOptions:

test {
    options.suiteXmlFiles = ['suite1.xml','suite2.xml']
}

This can of course be changed to options.xmlSuite(...). Not sure if we should add xmlSuite(...) on Test otherwise Test will have functions for multiple test-frameworks.

The gradle clean test followed by a gradle test was indeed an issue. I forgot to remove the temporary suite file before creating it again.

I'm also in favour of using the repositories in stead of the patch files but how do you want me to share my work with you then?

Clumsy was possible an inappropriate adjective on my part, I perhaps should have said verbose.

The suiteXmlBuilder is probably a very good way of replacing the suite XML files, we should keep it. There are some complex, multi-layer testing structures that using the suites brings. That flexiblility is worth hanging on to.

I hadn't spotted the suiteXmlFiles, so thanks for pointing that out. Of course the whole point is to get rid of XML but people may already have the suite XML files so it is good that they can be used.

I guess what I should have said is that the features you can use within a testng tag in the Ant build files and the ability to speicify includes directly in a POM are not presented yet in teh TestNG Gradle plugin. I guess if we can provide this capabilities in addition to the suite capabilites then things are looking really good.

Aha, I'm glad you have a solution to the clean test / test problem.

Regarding repositories I think the best bet is for you to start a branch on Launchpad – or some other place if you prefer. Basically DVCS relies on people publishing their branches: DVCS is pull technology where CVCS is push technology If you do start a branch on Launchpad I will subscribe to it and then merge in changes into my branches.

This is kind of pet peeve: can we use something other than 'similarly-named build.gradle' ?
When the source code base includes files with spaces, it ends up being unfriendly
to unix folks who like to say stuff on the command line like: find . | xargs grep ...
Unadorned commands tend to get confused when tokenizing stuff like this.

While there are of course ways of quoting and awk-ing your way out of the files-with-spaces
hassle on unix, it's still...well... a hassle!

I'd love it if we could have an internal-whitespace-free set of filenames in Gradle's source tree.
Is there a reason why we actually need a file with a space in its name?

Yes - to test that we deal with files with whitespace in their name