[GRADLE-3189] TestNGListenerAdapterFactory equality is flawed -- makes testing unusably slow Created: 29/Oct/14  Updated: 25/Jan/17  Resolved: 18/May/16

Status: Resolved
Project: Gradle
Affects Version/s: None
Fix Version/s: 3.0-milestone-1, 3.0-rc-2

Type: Bug
Reporter: Gradle Forums Assignee: Paul Merlin
Resolution: Fixed Votes: 5


 Description   

So I'm not fully familiar with the gradle source so a lot of this was from inspection, so take some of the details with a grain of salt. Also not a gradle expert so I may have just screwed up the basic invocation.

But basically, I have about 700 testng unit tests and tens of thousands of classes that I want to run with gradle. I also have a TestListener that I would like to attach. So from the docs, that looks like:

task testng(type: Test) {
useTestNG() {
listeners << 'testing.SingletonEnforcer'
suites file('testng/unit/config/unit.xml')
}

testClassesDir = sourceSets.testng.output.classesDir
classpath = sourceSets.testng.runtimeClasspath
workingDir 'testng/unit/config'
}

However, if I do this and have both listeners and suites and declared above, it seems like I never get to actually running my testng tests – I let it run overnight with no luck. So I fired up JProfiler and checked out where all the time was going and [it seems to be]([1]https://www.dropbox.com/s/j25lsab01ni...) in TestNGListenerAdapterFactory. Specifically the [invoke method]([2]https://code-review.gradle.org/browse...) at the bottom of the class which is using reflection to call... something... I don't know enough about the codebase to know what it's wrapping but my best guess is that it's wrapping every class in the codebase? Or maybe my class is interacting poorly with gradle's testng and now is wrapping itself over and over again? Is there any hints someone could give as to what I could even begin doing to debug this? I'm thinking I may compile my own gradle with some more debugging information to find out what classes it's wrapping tomorrow but figured I'd post for right now and see if anyone has any ideas. Here's a picture of the linked HTML output just for ease of access: [3]

P.S. The jprofiler was run with method sampling so I may have just been "unlucky" but 233 seconds of unlucky seems a little extreme to me
----------------------------------------------------------------------------------------
[1] https://www.dropbox.com/s/j25lsab01nirqgv/Hot_Spots.html?dl=0
[2] https://code-review.gradle.org/browse/Gradle/subprojects/plugins/src/main/groovy/org/gradle/api/internal/tasks/testing/testng/TestNGListenerAdapterFactory.java?hb=true#to67
[3] https://d2r1vs3d9006ap.cloudfront.net/s3_images/1109800/Selection_005.png?1413512337

 Comments   
Comment by Gradle Forums [ 29/Oct/14 ]

I just realized the class hierarchy of `SingletonEnforcer` might be useful since it's doing reflectiony stuff. The class just extends [`IExecutionListener`]([1]http://testng.org/javadoc/org/testng/...). I notice that this isn't mentioned in the documentation of [listeners]([2]http://www.gradle.org/docs/current/gr...), which says:

> Fully qualified classes that are TestNG listeners (instances of org.testng.ITestListener or org.testng.IReporter).

`IExecutionListener` looks like a sister class to `ITestListener` and not an instance of it. Could this be what's causing this?
----------------------------------------------------------------------------------------
[1] http://testng.org/javadoc/org/testng/IExecutionListener.html
[2] http://www.gradle.org/docs/current/groovydoc/org/gradle/api/tasks/testing/testng/TestNGOptions.html

Comment by Gradle Forums [ 29/Oct/14 ]

So I've exhausted what I can do with it. It looks like it's trying to wrap the object in itself or rewrap the object over and over again, or something like that. I tried parsing what the code was doing but it's a little too much java-reflection-metaprogramming-magic for me. Does anyone have any thoughts on this? Should I move this into a bug?

Comment by Gradle Forums [ 29/Oct/14 ]

Could you perhaps provide a self-contained example project that allows to reproduce the problem? This would be very much appreciated.

Comment by Gradle Forums [ 29/Oct/14 ]

Hmm, so i spent a good bit of time doing this and failed . So the issue is definitely more complex than I outlined above and/or I'm doing something wrong. Is there any way to get some more debugging out of the gradle test executor to see what it's doing? It seems like the task launches that java process and it black-boxes.

Also, thank you for the response!

Comment by Gradle Forums [ 29/Oct/14 ]

Ah. so i've figured out what was actually adding the issues. It seems to be that the reflection is kicked off in some strange way based on the number of `.xml` suite files that are included. For example my tests work fine if there are 11 `.xml` files in 24.472 seconds. When I add one more xml file, which itself only adds one class, it goes to 47s. adding one more takes it to 1.5m, etc. I've reproduced this successfully using a toy project that just generates a bunch of class files. Much of the project is unnecessary now that I know the issue is probably around the xml files and not the actual class files but... you know... i was experimenting . I've put it up on github at: [1]https://github.com/linkinpark342/grad... which generates 100 xml files and never completes. Obviously given the numbers I'm calling out above, 100 is COMPLETE overkill. I'll try to put up a branch that does fewer xml files but need to get back to work at this point XD

This is an issue because our "unit.xml" actually includes a bunch of subtests (so they can be run individually for development).
----------------------------------------------------------------------------------------
[1] https://github.com/linkinpark342/gradle_slow_testng.git

Comment by Gradle Forums [ 29/Oct/14 ]

This sounds like an actual bug then, I'm going to move this to the bug tracker (once i find where it is)

EDIT: guess I'm not – looks like the jira is locked down from creating new issues

Generated at Wed Jun 30 12:42:20 CDT 2021 using Jira 8.4.2#804003-sha1:d21414fc212e3af190e92c2d2ac41299b89402cf.